During my ongoing work on the OrangeLine DEJAVU module, I stumbled over the question which part of the module state should be held just as member variables of my module or better held in JSON storage to be restored after a patch (re)load or preset load.
The general question to be answered is whether it is expected that a module continues to work after exiting VCV Rack and reopening or saving and reloading the patch as if this never would have happened.
The consequences are quite important:
If answer to that question is NO, then continuing a performance after interrupting it, is not really possible. Wearing the hat of a perfectionist, I would like to have a patch being able to continue just from the point I closed VCV Rack after restarting it.
If the answer is YES, then any use of a member variable of a module might be a no go. If a variable would be necessary to carry information of any kind of state between one process() call to the next one, this state is lost after exiting and reloading, compromising the continuation of the patch. Otherwise, the member variable could just be a local variable of the the process() method.
Holding all state information in JSON storage will for some modules drastically increase the amount of data to save and restore when saving, and loading a patch. Especially the auto-save might impact the performance of Rack as a whole. Also to file size of patches and presets might be increased quite a lot.
As an example:
DEJAVU holds 16 x 2 + 4 + 1 = 37 pseudo random generator states. The pseudo random generators using a proven mathematical model developed by smarter guys than me but each one holds state information in an array of 624 unsigned longs. Storing this state information with JSON would just use up 8 x 624 Bytes which about 5K. 5K times 37 would give a file size of nearly 200K for storing one DEJAVU state. I can think about some workaround to reduce it but this would cost me quite a lot of work and maybe performance penalties.
I’m really interested how other developers out there are thinking about this topic in general.
I haven’t surveyed the field, but my general sense is that (pseudo-)random patches aren’t expected to resume in exactly the same state that they stopped in (which would of course only be strictly possible if all modules agreed to do so, and I think many don’t–LFOs and oscillators typically restart at zero, right?) I don’t know anything about DEJAVU other than that it looks neat, so I don’t know if its own state would be particularly meaningful to restore even if other modules restored in different states.
That said, I’m not sure 200K/module is impractical, even in V1 (V2 promises a different local storage model which should make this easier). I’d be interested to know if dumping the state actually bogs things down very much.
You’ve probably thought of the workaround which stores an initial seed (from which the starting state can be deterministically derived) plus the number of times each generator has been polled, and then seeds the generators and runs them forward on load, blocking until they’re caught up. You’re trading arbitrarily slow start times for small patch sizes. If your module is polling the generator every sample, this could get prohibitively long, but if it’s polling on a clock division, and the patch was saved after a typical runtime, that might be more workable. You could bound the runtime problem by forcing a deterministic re-seed after a certain amount of clock time, which would also reset the polling counters.
Whatever you decide, you could have it as a right-click option, with the default probably off; that way people who want state saved and are willing to make some tradeoff for it can do so.
Just save your whole buffer in one JSON string with string::toBase64() and fromBase64(). Rack 2 will support patch file assets so you can save this data to an arbitrarily large file, but 184KB is fine in JSON for Rack 1.
Hi,
thanks for your proposal. Already thought about that and added the number of getRandom calls to the random generator struct to prepare for this option. And yes storing the big state, or stepping up using the getRandom call count, or accepting non continuous behavior could be a right click option. I can do all of that but it does not make sense if a patch is not expected to be continuous at all. It would be work for nothing if all other modules don’t care about this and break the patch after reload anyway. That’s why I put it here as a general question.
Regards
Dieter
If you don’t have some kind of philosophical reason to use to such generators, “normal” musical use doesn’t really require a random generator that has hundreds of bytes of state, even for generating audio rate noise. You might get by with 8 bytes or so per generator to get results that are sufficiently random. (I personally prefer using Mersenne Twister, that has over 2000 bytes of state, but that’s simply because of those “philosophical” reasons, not really for any practical reason.)