I will have a presentation and talk at the 9. International Modelica Conference in Munich at 3.-5. September 2012. It will focus on:
- ncDataReader2 – my file reader library,
- its usage from Modelica and
- its application in recent research projects.
Version 2.3.0 is out now, but I hope to get some improvements done til september. The paper and slides of the talk will be linked here when available.
Version 2.3 of my data reader library is coming in the next days. Here’s a list of recent improvements and bug fixes:
- added CosWin approximation
- added constant extrapolation
- example GenerateBigFile is much faster now
- fixed a bug with Akima and default extrapolation near the right border that existed for a long time
- added functions to dump access statistics, useful to optimize paramters and caches
- added error dialog for Win32: now you get error mesages within Dymola and other software that hides stdout from the user
- added command line client ncdr2Dump1D to dump interpolated values, very useful for testing
- build with netCDF 4 on Windows including DAP supported
- documentation updated to include the new features
A source package and binary package for Windows will be uploaded to GitHub within the next days.
When preparing files for the ncDataReader2 I often have to convert text files that contain values separated by TABs or commas. In most simple cases this can be done with some lines of python code. An example is here:
import sys, pupynere
ll = [l.strip().split(',') for l in open(sys.argv) if not l.startswith('#')]
vv = zip(*[map(float, l) for l in ll])
nc = pupynere.netcdf_file(sys.argv+'.nc', 'w')
for i in range(len(vv)):
nc.createVariable('var_%02d' % i, 'd', ('dim',))[:] = vv[i]
This short program will read the column data separated by commas, skip comment lines starting with # and write a netCDF file, all in 7 lines of code! Pupynere is one of the netCDF-modules available for python. It is written in pure python and has an API which is compatible with ‘Scientific.IO.NetCDF’. This simple approach works for small, well-formed files without headers. For all other cases I’m working on more sophisticated tool which uses the CSV module of python.
A new release 2.2.4 of ncDataReader2 is out. Some bugs were fixed and the API has slightly changed. This version should work both with Dymola and OpenModelica on Linux and Windows. When you have a DAP-enabled netcdf library it will even fetch data from the web in background. I already did some Dymola simulations with on-demand loading of data from a server, a post about this topic will follow.
- the meaning of offset and scale factor for a ncDatSet1D was changed
- chunk loading was implemented
- Akima interpolation was changed heavily, works much faster now
- better support for newer netcdf libraries
- new helper functions for setting attributes
- changed some char* to const char* to work with OpenModelica
- a lot of bugs were fixed
A new minor release of ncDataReader2 is coming. It only fixes one important bug:
- Numerical attributes (like scale_factor and add_offset) were not read correctly with 2.2.0.
Source and binary packages will follow soon.
This example demonstrates the use of the library ncDataReader2 to access data sets from Modelica in a way suitable for the simulation system. The access is handled via external functions from Modelica which read and interpolate the data stored in netCDF files.
Dymola is used as the simulation system here. As ncDataReader2 is a simple, platform-independant library implemented in ANSI-C this approach should work (in theory) with other Modelica implementations too. As soon as I have tested other simulation systems the results will be documented here.
When working with complex models you often have to access external data. This may be measured time rows of climate parameters or friction factors for different plant components. Some advantages of using ncDataReader2 in such situations:
- data is presented as continous functions of one or two variables with automatic interpolation
- can handle very large data sets that will not even fit into memory at once (load on demand)
- offers different interpolation and extrapolation methods
- data is stored in netCDF format which offers very efficient access to multi-dimensional data
- data files can be easily exchanged without changing or recompiling your model
- data files can contain many different data sets (scalar, 1D, 2D) at the same time
- when you store interrelated data sets in one file, the chance of accidently mixing inconsistent data is minimized
A typical use is reading weather files into a thermal building simulation. The weather files contain different climate parameters like air temperature and humidity, radiation, wind speed as time rows for one year as well as latitude, longitude and the timezone for one location. You can easily exchange the data file and simulate the building in a totally different location without touching the model. ncDataReader2 is used in the Modelica library FluidFlow.
If you don’t already have Dymola installed, you can download a demo version here. It has some restrictions but will work with the simple model used in this example.
The simple model
The following model is used as a starting point for our example. The equation for t will be exchanged with a function call in the next steps. x is just a dummy to have something to integrate.
There is no special installation procedure. Just put the necessary header and library files into the working directory (usually where you saved the model file). Which files are actually needed depends on the Dymola version and the compiler (gcc, msvc). The easiest way is to extract the whole contents of precompiled binary package (Win32 zip) into the working directory. It includes everything needed by Dymola and the executables ncgen and ncdump which are used in the next step. If you compile ncDataReader by yourself, you should at least copy ncDataReaderEA.h and the netCDF and ncDataReader2 library files (dll and/or lib).
Defining Modelica functions
The next file makes the C functions of the library ncDataReader2 available in Modelica. It contains descriptions of the most important functions of the library (actually of the easy api / EA) that can be read by Modelica. Just put it in the working directory.
Generating the data file in netCDF format
But in this example we will write a CDL text file and convert it to netCDF. This step does not require additional software. Use the command:
ncgen -o datafile.nc datafile.cdl
to convert the following file or download the example file here. ncgen is usually shipped with netCDF.
This file contains the two variables time and temperature . The temperature will be interpolated with cubic Akima splines and time will be handled periodically. A netCDF file can be converted back to a CDL file with ncdump.
Adjust the model and simulate
The variable t as a function of time will now contain values from the file instead of the functions used above. An import statement is necessary to use the functions from the library. The call to NcEasyGet1D will return a value of the variable temperature in the file datafile.nc for the current value of the variable time.
That’s it! Just run the simulation and look at the results.
The temperature is read and interpolated from the file. It can be used as a normal variable in models.
This was only a very basic example. Of course you can read and interpolate a large number of variables using different methods of interpolation and extrapolation. Scalar parameters can be read as well. Please have a look at the documentation of ncDataReader2 for more information on the naming conventions, attributes and more.
The conversion of data to the netCDF formats looks confusing to the beginner but is not hard to understand. A typical CSV file can usually be converted with a couple of lines of Python code which is my favourite method.
ncDataReader2 works both with netCDF-3 and netCDF-4, but you should not mix the tools and libraries between both versions.
A new release of ncDataReader2 is out. Changes include:
- switched from autoconf to cmake
- tested to compile on Linux, Windows and MacOS
- the easy API (EA) was completed
- EA works now with Modelica (Dymola):
- a weather file reader was implemented and tested with the modelica library FluidFlow
- new documentation in english
- support for netCDF attributes was completed