pystencils issueshttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues2024-03-27T17:25:53+01:00https://i10git.cs.fau.de/pycodegen/pystencils/-/issues/88Clarify semantics of fancy integer division functions2024-03-27T17:25:53+01:00Frederik HennigClarify semantics of fancy integer division functionsThe functions `modulo_floor`, `modulo_ceil`, `div_floor` and `div_ceil` of the `integer_functions` exhibit unclear rounding behaviour. Their names and docstrings indicate mathematical rounding behaviour ("down" is negative infinity), whi...The functions `modulo_floor`, `modulo_ceil`, `div_floor` and `div_ceil` of the `integer_functions` exhibit unclear rounding behaviour. Their names and docstrings indicate mathematical rounding behaviour ("down" is negative infinity), while their implementation performs zero-oriented rounding ("down" is toward zero), as this is the default behaviour of C `/` and `%`.
We should clarify the semantics of these functions and adapt docstring, implementation, or both.
See also the discussion here: pycodegen/pystencils!368Release 2.0Daniel BauerDaniel Bauerhttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/87Symbolic Language2024-03-27T17:25:53+01:00Frederik HennigSymbolic LanguageThe new backend must support the full symbolic language that pystencils offered previously, plus some sensible additions. For now, however, the focus should be on implementing code generation support for all previously supported SymPy fe...The new backend must support the full symbolic language that pystencils offered previously, plus some sensible additions. For now, however, the focus should be on implementing code generation support for all previously supported SymPy features, as well as everything in `pystencils.sympyextensions` and a few other submodules that add features to kernels.
## Symbolic Language Consolidation
All symbolic language features should be collected into the `pystencils.sympyextensions` module (as also discussed in pycodegen/pystencils#47).
Furthermore, the documentation should include a section that fully documents all SymPy features and our extensions which the code generator supports. This section should also point out any restrictions and caveats.
- [ ] Set up symbolic language documentation
## FreezeExpressions
The language of pystencils will basically encompass everything that `pystencils.backend.FreezeExpressions` can translate.
Various pystencils features are still unimplemented:
- [ ] Augmented Assignments
- [ ] AddressOf
- [ ] Relations (sp.Relational)
- [ ] pystencils.sympyextensions.integer_functions
- [x] Bitwise ops
- [x] Integer Division
- [ ] Fancy division operations (see pycodegen/pystencils#88)
- [ ] pystencils.sympyextensions.bit_masks
- [ ] GPU fast approximations (pystencils.fast_approximation)
- [ ] ConditionalFieldAccess
- [ ] sp.Piecewise
- [ ] sp.floor, sp.ceiling
- [ ] sp.log, sp.atan2, sp.sinh, sp.cosh. sp.atan
- [ ] sp.Min, sp.Max: multi-argument versions
- [ ] Modulus (sp.Mod)
- [ ] Folding functions (`sp.Sum`, `sp.Product`, see pycodegen/pystencils#55)
## Random Number Generation
The random number generators currently implemented in `pystencils.rng` must be integrated with the new backend, in particular with `FreezeExpressions`.
- [ ] Integrate RNGs
## Control Flow Structures
With the removal of the old `astnodes` module, there is currently no way to express control flow in the front-end / the symbolic language.
We should discuss whether, and how, we should introduce loops and conditionals into that language again.https://i10git.cs.fau.de/pycodegen/pystencils/-/issues/86Migrate test cases & notebooks2024-02-28T11:48:46+01:00Frederik HennigMigrate test cases & notebookshttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/85Staggered Kernels2024-03-15T09:55:35+01:00Frederik HennigStaggered KernelsThe code generator for staggered kernels must be re-implemented using the new backend.The code generator for staggered kernels must be re-implemented using the new backend.https://i10git.cs.fau.de/pycodegen/pystencils/-/issues/84CPU Platforms and Transformations2024-03-15T10:39:04+01:00Frederik HennigCPU Platforms and Transformations# CPU Recognition and JIT
- [ ] Implement CPU auto-recognition for the `CurrentCPU` target
- [ ] Re-introduce field size equality checks to the CPU JIT compiler
- [ ] Replace the legacy CPU just-in-time compiler with an object-orient...# CPU Recognition and JIT
- [ ] Implement CPU auto-recognition for the `CurrentCPU` target
- [ ] Re-introduce field size equality checks to the CPU JIT compiler
- [ ] Replace the legacy CPU just-in-time compiler with an object-oriented, modular structure
# CPU Optimizations
The sequence of optimizations applied to a CPU code shall be controlled and carried out by the CPU optimization driver `pystencils.backend.kernelcreation.cpu_optimization.optimize_cpu`. Each individual optimization pass shall be implemented in a dedicated class within the `pystencils.backend.kernelcreation.transformations` module.
## Loop Optimizations
- [ ] OpenMP
- [ ] Loop-Invariant Code Motion: Fixpoint analysis and transformer to extract loop-invariant code
- [ ] Loop Cutting and condition elimination
- [ ] Loop tiling / blocking
## Vectorization
Vectorization shall be implemented as a two-step procedure.
### Generic Vectorization
In the first phase, the generic vectorizer shall analyze the target loop's body. If it is vectorizable, it shall be vectorized by adapting all data types to their respective vector types; adapting the loop range and, if necessary, create a remainder loop; and replace array accesses with vectorized array accesses.
- [ ] Implement vectorization legality analysis
- [ ] Implement vector transformation
### Intrinsics Selection
The second phase is intrinsics selection. The vectorizer calls upon the platform, which must be a subclass of `GenericVectorCpu`, to map all vectorized operations onto intrinsic functions. The intrinsic selection pass shall be implemented in `pystencils.backend.transformations.select_intrinsics.MaterializeVectorIntrinsics`. This class interacts with the protocols defined in `pystencils.backend.platforms.GenericVectorCpu` to retrieve the actual intrinsics depending on the vector architecture.
- [x] Implement protocols to select vector type, constant, load, store, and arithmetic intrinsics
- [ ] Implement protocols to select variable broadcast intrinsics
- [ ] Implement protocols to select mathematical function intrinsics (e.g. trigonometry, transcendentals, ...)
### X86
Intrinsic selection for X86 is implemented in `pystencils.backend.platforms.x86`:
- [x] vector types
- [x] constants
- [x] arithmetic intrinsics
- [x] aligned and unaligned load/store
- [ ] gather/scatter
- [ ] variable broadcast
### ARM
Intrinsic selection for ARM shall be implemented in `pystencils.backend.platforms.arm`:
- [ ] vector types
- [ ] constants
- [ ] arithmetic intrinsics
- [ ] aligned and unaligned load/store
- [ ] gather/scatter
- [ ] variable broadcasthttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/83GPU Platform2024-03-15T09:54:26+01:00Frederik HennigGPU Platform## Kernel creation for GPUs
- [ ] Implement CUDA GPU platform into `pystencils.backend.platforms`
- [ ] Implement GPU indexing and iteration space materialization (might want to re-use the existing `GpuIndexing` classes)
## Parameter...## Kernel creation for GPUs
- [ ] Implement CUDA GPU platform into `pystencils.backend.platforms`
- [ ] Implement GPU indexing and iteration space materialization (might want to re-use the existing `GpuIndexing` classes)
## Parameterization
- [ ] Implement automatic GPU detection for the `CurrentGPU` target
- [ ] Add GPU options to `CreateKernelOptions`
## JIT
- [ ] Implement CUDA just-in-time compilation into `pystencils.backend.jit`https://i10git.cs.fau.de/pycodegen/pystencils/-/issues/75Modernize pystencils' dev environment2024-01-09T14:10:28+01:00Frederik HennigModernize pystencils' dev environmentWe should modernize the development environment of pystencils to latest Python practice. This includes:
### pyproject.toml
We should migrate from the `setup.py` paradigm of `setuptools` to the now-standard `pyproject.toml` ([PEP 518](h...We should modernize the development environment of pystencils to latest Python practice. This includes:
### pyproject.toml
We should migrate from the `setup.py` paradigm of `setuptools` to the now-standard `pyproject.toml` ([PEP 518](https://peps.python.org/pep-0518/)) for the basic build system configuration.
This way, we can enjoy all the helpful tools that now base themselves on `pyproject.toml` (see 'Dependency Management' below).
Still, some tool-specific stuff may have to remain in `setup.py`.
### Dependency Management
Currently, as far as I am aware, setup of a dev environment for pystencils depends on `requirements.txt` files flying around wildly.
To consolidate and clearly define a uniform development environment. we should instead introduce a packaging and dependency manager into the project - I'd recommend [pdm](https://pdm-project.org/latest/). This way, all developers would get the exactly same environment, with exactly the same versions of all required dependencies - including dev and build dependencies like `flake8`, `sphinx`, etc.
### Type Checking
pystencils has become a vast package with many moving parts, and as a generator of exectuable code, a high level of correctness is expected of it. While extensive unit tests are a necessary tool to achieve correctness, they can never be entirely watertight. Being written in Python, users and developers of pystencils enjoy great flexibility due to Python's dynamic type system. However, that same type system may also permit a much larger number of (potential) bugs to go unnoticed for a very long time.
To alleviate this issue, Python itself has been gradually introducing and improving its system of type hints, which although are not mandatory
nor do they affect the code in any way by themselves. However, type checkers like [mypy](https://www.mypy-lang.org/) have been developed on top of that to statically check the types in Python code.
In particular in the context of pycodegen/pystencils#73, we should gradually introduce thorough static type checking into pystencils.
This will help to make the code more robust and achieve a higher level of correctness a-priori. Types can be checked on developer's machines (e.g. as a hook to `git add`/`git commit`) and in the CI (much like linting with `flake8`).https://i10git.cs.fau.de/pycodegen/pystencils/-/issues/74Constness in generated code2024-03-27T18:23:27+01:00Frederik HennigConstness in generated codeThe way pystencils determines constness and placement of `const` keywords is very intransparent and at times misleading.
Things that should be const are often printed non-const. While we may expect compilers to detect constness by themse...The way pystencils determines constness and placement of `const` keywords is very intransparent and at times misleading.
Things that should be const are often printed non-const. While we may expect compilers to detect constness by themselves, this
makes integration into other code frameworks more tedious, and still may impact performance.
Luckily, so far, I have not observed the opposite (non-const values printed const) which would just not compile.
## Constness
Constness is introduced in the following places (maybe I've missed something):
- In the type system:
- `BasicType` and `StructType`: An object type may be `const`-qualified. That is how it should be in a C code generator.
- `PointerType`: A pointer may be `const`-qualified, and its pointed-to type may also be; so theoretically, all four possible combinations of constness on pointers can be realized. This does not hold for the recently introduced double pointers (pycodegen/pystencils!356), though; see below.
- In the AST: `SympyAssignment` has a member `is_const` for constant declarations.
While it is perfectly sensible to const-qualify types, annotating assignments with context has the potential for vast confusions; what if the type of the LHS is non-const? How to print it if it is const?
## Problems
### Field Access Resolution
In `transformations.py` ([here](https://i10git.cs.fau.de/pycodegen/pystencils/-/blob/master/pystencils/transformations.py#L557)), the data pointer for a field access is created with `const=True` if the field is read-only; however this creates a const pointer, not a pointer to const. The pointed-to data type is still non-const.
### Code Printer
The code printer handles const `SympyAssignment`s ([here](https://i10git.cs.fau.de/pycodegen/pystencils/-/blob/master/pystencils/backends/cbackend.py#L273)) in a fashion that is clearly problematic: The LHS type is printed, and then all occurences of `const` are removed from it by string operations. Then a leading `const` is added.
This breaks down in the presence of pointers (especially now, with the new double pointers): `const` may occur more than once in a pointer type. Also, if the declaration LHS is of pointer type, the `const` keyword must not be added to the left, but to the right of the type string.
## Solutions
Constness should exclusively be a part of the type, and the type system should respect it.
I've tried several avenues of fixing this, without success; even though constness only occurs at very few places in the code, the way they are spread across the code makes changing them extremely challenging.
The constness issue should enter consideration of the design of the new backend (pycodegen/pystencils#73), where it can then be solved as a 'by-the-way', simply by clean integration of constness in the type system.Release 2.0Frederik HennigFrederik Hennighttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/73pystencils Backend Rework2024-03-20T10:53:56+01:00Frederik Hennigpystencils Backend ReworkThe code generation backend of pystencils is due for a major overhaul. This issue is meant to track the efforts in this direction.
Development branch: `pystencils/backend-rework`
The various sub-tasks of this project are documented usi...The code generation backend of pystencils is due for a major overhaul. This issue is meant to track the efforts in this direction.
Development branch: `pystencils/backend-rework`
The various sub-tasks of this project are documented using the Tasks below.
Documentation for this branch is currently served here: https://da15siwa.pages.i10git.cs.fau.de/dev-docs/pystencils-nbackend/Release 2.0Frederik HennigFrederik Hennighttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/60SymPy CSE does not work as good as it used to work2022-03-27T15:17:16+02:00Markus HolzerSymPy CSE does not work as good as it used to workhttps://github.com/sympy/sympy/commit/19c715920e52337d124dda63abdc056f56ec094e
has a negative impact on our CSE routine. An example would be `test_simplifications_srt_d2q9_incompressible` in lbmpy
The following example should demonstra...https://github.com/sympy/sympy/commit/19c715920e52337d124dda63abdc056f56ec094e
has a negative impact on our CSE routine. An example would be `test_simplifications_srt_d2q9_incompressible` in lbmpy
The following example should demonstrate the change, before 19c715920e52337d124dda63abdc056f56ec094e:
```python
import sympy as sp
var = sp.symbols("a b c d")
test = -var[0] + var[3]/9 + var[2]**2/2 + var[2]/3, -var[1] + var[3]/9 + var[2]**2/2 - var[2]/3
sp.cse(test)
>>> ([(x0, c/3), (x1, c**2/2 + d/9)], [-a + x0 + x1, -b - x0 + x1])
```
After 19c715920e52337d124dda63abdc056f56ec094e:
```python
import sympy as sp
var = sp.symbols("a b c d")
test = -var[0] + var[3]/9 + var[2]**2/2 + var[2]/3, -var[1] + var[3]/9 + var[2]**2/2 - var[2]/3
sp.cse(test)
>>> ([(x0, c/3), (x1, c**2)], [-a + d/9 + x0 + x1/2, -b + d/9 - x0 + x1/2])
```
The second latest master version does not find the replacements for the divisions.
An issue in Sympy was created here: https://github.com/sympy/sympy/issues/23297Markus HolzerMarkus Holzerhttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/57Clearer Syntax for pystencils Field2022-02-26T19:14:55+01:00Markus HolzerClearer Syntax for pystencils FieldAt the moment this is equivalent:
```
field = ps.fields(f"field(2): [2D]")
field[0, 0](1) == field.center(1) == field(1)
```
On the one hand side having this flexibility can be nice on the other it can lead to a lot of confusion if al...At the moment this is equivalent:
```
field = ps.fields(f"field(2): [2D]")
field[0, 0](1) == field.center(1) == field(1)
```
On the one hand side having this flexibility can be nice on the other it can lead to a lot of confusion if all of the above forms are mixed throughout the package. I would propose to only stick to `field[0, 0](1)` and maybe even depreciate all others. It is the most explicit form.Markus HolzerMarkus Holzerhttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/55Support Reductions2024-03-15T14:29:57+01:00Markus HolzerSupport Reductionspystencils supports sympy Sum like this:
`sum = sp.Sum(sp.abc.k, (sp.abc.k, 1, 100))` will be printed as:
```c++
{
for (int64_t ctr_0 = 0; ctr_0 < _size_x_0; ctr_0 += 1)
{
_data_x[_stride_x_0*ctr_0] = [&]() {
doubl...pystencils supports sympy Sum like this:
`sum = sp.Sum(sp.abc.k, (sp.abc.k, 1, 100))` will be printed as:
```c++
{
for (int64_t ctr_0 = 0; ctr_0 < _size_x_0; ctr_0 += 1)
{
_data_x[_stride_x_0*ctr_0] = [&]() {
double sum = (double) 0;
for ( int k = 1.0; k <= 100.0; k += 1 ) {
sum += k;
}
return sum;
}();
}
}
```
This is C++ code and additionally, at the moment, it bypasses the type system a bit. Thus this should be reimplemented.Release 1.1https://i10git.cs.fau.de/pycodegen/pystencils/-/issues/53Show Assembly Instructions2022-06-29T09:47:44+02:00Markus HolzerShow Assembly InstructionsLike `ps.show_code` but to get the assembly instructions. Like #28 but without KernCraft.Like `ps.show_code` but to get the assembly instructions. Like #28 but without KernCraft.Release 1.1Markus HolzerMarkus Holzerhttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/52Add possiblity to create ".pvd"-files in vtk-writer2022-06-29T09:47:56+02:00Christoph SchwarzmeierAdd possiblity to create ".pvd"-files in vtk-writerCurrently, the files that are written in pystencil's vtk-writer are stored as _vtkImageData_ (`.vti`) only. It would be great if the vtk-writer was able to create a file in _ParaView Data format_ (`.pvd`) which contains the path to each ...Currently, the files that are written in pystencil's vtk-writer are stored as _vtkImageData_ (`.vti`) only. It would be great if the vtk-writer was able to create a file in _ParaView Data format_ (`.pvd`) which contains the path to each `.vti`-file and the (LBM) time step at which the file was written.
In ParaView, "time" would then actually represent the (LBM) time step instead of being equal to the index, i.e., number of the loaded ".vti"-file.Release 1.1Markus HolzerMarkus Holzerhttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/51FAQ2021-12-10T13:37:01+01:00Markus HolzerFAQA FAQ in the repo would be good to have to list pitfalls etc which get reported. A first point to list for example could concern the installation of pystencils:
From time to time problems arise when installing `pystencils`. The latest ...A FAQ in the repo would be good to have to list pitfalls etc which get reported. A first point to list for example could concern the installation of pystencils:
From time to time problems arise when installing `pystencils`. The latest problem for example was that `pip` was not installed in a new `conda` environment and then for compiling the boundary C-File was linked with the wrong python version. Problems like these should be collected in an FAQ file or something similar.
Just by comparing with `NumPy` there is quite a long install prerequisite file although the installation of `NumPy` is really straightforward. Something like this I already imagine for `pystencils`:
https://numpy.org/install/Markus HolzerMarkus Holzerhttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/50CustomCodeNode should be treated like SympyAssignment in transformations (cur...2021-11-29T19:31:41+01:00itischlerCustomCodeNode should be treated like SympyAssignment in transformations (currently RNG numbers are calculated too often)I don't know if it is a bug or if I am using it wrong, but when using mutliple rng symbols via `next` on e.g. fluxes in different directions, the generated code will calculate each random number when only one is used. so basically for a ...I don't know if it is a bug or if I am using it wrong, but when using mutliple rng symbols via `next` on e.g. fluxes in different directions, the generated code will calculate each random number when only one is used. so basically for a D3Q19 stencil Philox is called 5 times more then is needed, which slows down the program drastically.
```
import pystencils as ps
import sympy as sp
from pystencils.rng import random_symbol
# Setup
stencil = 19
L=(10,10,10)
dh = ps.create_data_handling(domain_size=L, periodicity=True)
c_field = dh.add_array('c', values_per_cell=1)
j_field = dh.add_array('j', values_per_cell=stencil // 2,
field_type=ps.FieldType.STAGGERED_FLUX)
def grad(f):
return sp.Matrix([ps.fd.diff(f, i) for i in range(dh.dim)])
# genergate flux eq. and set them to 0
fvm_eq = ps.fd.FVM1stOrder(c_field, flux=grad(c_field))
flux=[]
for diff in fvm_eq.discrete_flux(j_field):
flux.append(ps.Assignment(diff.lhs, 0))
flux = ps.AssignmentCollection(flux)
# define rng
rng_symbol_gen = random_symbol(flux.subexpressions, dim=dh.dim)
for i in range(len(flux.main_assignments)):
fluct = next(rng_symbol_gen)
flux.main_assignments[i] = ps.Assignment(flux.main_assignments[i].lhs, fluct)
print(flux)
flux_kernel = ps.create_staggered_kernel(flux).compile()
ps.show_code(flux_kernel)
```Markus HolzerMarkus Holzerhttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/47Clean extended SymPy functions2024-03-27T18:21:34+01:00Markus HolzerClean extended SymPy functionsThere are some functions which extend a SymPy function. These should all be implemented in a single module/fileThere are some functions which extend a SymPy function. These should all be implemented in a single module/fileRelease 2.0Markus HolzerMarkus Holzerhttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/46Vectorization revamp2024-03-27T18:21:42+01:00Jan HönigVectorization revampRelease 2.0https://i10git.cs.fau.de/pycodegen/pystencils/-/issues/45Modularise FD, Runhelper and others2021-11-19T15:54:46+01:00Markus HolzerModularise FD, Runhelper and othersFd and Runhelper should be modules that build upon pystencils. They should not be integrated directly in pystencilsFd and Runhelper should be modules that build upon pystencils. They should not be integrated directly in pystencilsMarkus HolzerMarkus Holzerhttps://i10git.cs.fau.de/pycodegen/pystencils/-/issues/42Logging2022-05-25T10:19:32+02:00Markus HolzerLoggingPython provides very good logging functionality which should be used in pystencils. For example problems in the type-system could be logged on a debug level. Oftentimes it is necessary to add print statements in a debug session to see wh...Python provides very good logging functionality which should be used in pystencils. For example problems in the type-system could be logged on a debug level. Oftentimes it is necessary to add print statements in a debug session to see which expressions are cast to which type etc. This could always be logged on a debug level so we can just always see what happens in the back.
https://docs.python.org/3/howto/logging.htmlMarkus HolzerMarkus Holzer