Create sound from image

Hello, I am trying to create a sound according to this. It seemed simple to me, but I’m making a mistake somewhere.

For the individual lines of the image, I will assign frequencies (now for the test only by linear interpolation), for example for line 1 it is 4000 and for the last 16000. Then I scan the image line by line where I add up the individual sinuses and save them in the sample. The resulting sound is completely different from the original.

for i=0, length - 2 do  -- length = duration * samplerate
        local t = i / length -- my time
        local x = floor(Lerp(0, bmp.width, t))  -- interpolation, bmp is image class
        local sum = 0 -- sum for all sines in column
        for y = 1, lines do  -- sum image columns
            local r, g, b, a = bmp:get_pixel(x, y - 1) -- get pixel color
            local amplitude = ((r + g + b) / 3) / 255 -- average brightness / 255 for 0 to 1 values
            sum = sum + Sine(t, amplitude, frequencyDistribution[y], 0) -- sum sines with distributed frequencies with 0 phase
        end
        sum = sum / lines -- average output value
        new_buffer:set_sample_data(1, i + 1, sum)
    end
function Sine(time, amplitude, frequency, phase)
    return amplitude * sin(2 * pi * frequency * time + phase)
end

what is wrong ? thanks.

1 Like

One thing wrong is the computation for “brightness”. See image - Formula to determine perceived brightness of RGB color - Stack Overflow for a pretty thorough discussion on computing various notions of brightness/luminance/value, and the underlying assumptions you may be making about the colorspace your data is in (linear vs gamma-corrected, etc.).

1 Like

this would be a perfect candidate for test driveN development. start with some simple test cases, like all black, one pixel lit, etc…

3 Likes

I am not sure if this helps, it used to be an active plugin for VCV 1.6 that processed images into cv. The code is still available and you might find it useful.

1 Like

That’s true, but this is enough for the test. First the problem and then everything around. I can control the amplitude with just the blue component of the color, for example. But good point :wink:

Thanks, I knew I had seen that module somewhere before. But that’s not exactly it. if I had to push it to the extreme, then VirtualANS by Alexander Zolotov is a model, but that will never happen :slight_smile:

Well there is this :

But it’s not as stable as it should be ?

yes, crash :frowning:

I really want to use that module with this:

…but it I don’t think they match up.

I am currently developing a module that takes an image and creates control signals – gates triggers and v/oct – from one or more “read heads”. I’m not focusing on generating audio, at least not in the first release, so it’s a bit of a different beast than described here.

It won’t be available for while, but I’m making slow progress amidst a bunch of competing tasks, like a kitchen remodel, water leaks in the house and the ferry to our island is on passenger-only service. etc ;-).

When it’s in good enough shape I’ll post a new thread asking for beta testers.

3 Likes

I knew there was a new module and could not find it

1 Like

OT: I did this using a raytracer that is not suitable at al for it ANS - IngoognI but it works.

With a static things like an image,you will obtain static harmonics

Making sound from video would have more sense

The left of the image is the left channel

The up is the high frequencies

…Or the human analyses images via a ‘scanline’ method…or via a spiral ‘polar’ method…or circle ‘concentric’ method…or anything else…sequentially

I think we’re veering away from the original question, so I’ll start a new thread to share what I’m working on in my “image to sound” module.

1 Like

Not necessary to start a new thread…It was just an inpiration

But if the project interest you…go on

“constellation interpolation”

A float polar pointer is navigating in a spiral manner into an integer pixel cartesian image

Blue is square

Red is sine

green is saw

Weight of pixels

intensity = frequency

Just an Idea…certainly not perfect…or completely wrong

Those are great ideas – and ones that could possibly work in the module I’m designing.

Design ideas for the module I’m building are here: #d - Imagine

2 Likes

I can only agree, I can think of many other things to use this potential :slight_smile:

1 Like

thanks for link.