mp3tag database not ready for prime time

Discussion:

John Ruttenberg

2002-02-13 12:07:09 UTC

Is it grinding at server startup? That would presumably be from refreshing
the genre hash, which is quick and dirty and expensive at the moment. It
iterates over the whole hash when it does that.

I can't really hear it. It's in a closet. Can we instrument it somehow?

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Tiny Wireless Camera under $80!
Order Now! FREE VCR Commander!
Click Here - Only 1 Day Left!
http://us.click.yahoo.com/nuyOHD/7.PDAA/yigFAA/rIp0lB/TM
---------------------------------------------------------------------~->

To unsubscribe from this group, send an email to:
slimp3-dev-unsubscribe-***@public.gmane.org

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

Sean Adams

2002-02-13 06:56:31 UTC

Permalink

A DBM is a fast hash on disk - it gives you gives you random r/w access.
We don't have to keep the database in RAM, or load it from disk when we
start up, or refresh the whole thing whenever we re-scan.

A text file can only provide random access if you keep an index, or the
records are in sorted order. Retrieveing a record from an unsorted text
file is O(n), ie you have to scan the whole file until you find what you
want. Even if it's sorted, a binary search O(log n) of a text file is
more expensive than a hash lookup at basically O(1), which is how dbm
files work. It's usually one, but seldom more than two disk hits per
lookup, because the index is cached.

Also, you can only write a text file at the end. If you want to insert a
record into a sorted text file, it's O(n) because you have to rewrite the
whole file after the point where you inserted the new record. You can
insert a record in a dbm any time you want, with cost O(1).

Finally, perl's interface for DBM is ingenious. It behaves just like a
normal perl hash, except it's synced to disk. All we have to do is use Tie
to connect our hashes to a file, and they become persistent. That's *it*!

Still, I want to bring this up again before we go too far down the path of
doing all this in the main server: we will never have correct multitasking
until this disk activity is moved to a separate process. Please keep that
in mind - we shouldn't spend too much time on this if it's not what we're
ultimately going to need.

Hmm.. it's only using 40MB. SDBM is not the best dbm library.
Berekely dbm and Gnu dbm are the best ones - no silly key/value length
restrictions, and no separate .dir and .pag files.

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Sponsored by VeriSign - The Value of Trust
When building an e-commerce site, you want to start with a
secure foundation. Learn how with VeriSign's FREE Guide.
http://us.click.yahoo.com/kWSNbC/XdiDAA/yigFAA/rIp0lB/TM
---------------------------------------------------------------------~->

To unsubscribe from this group, send an email to:
slimp3-dev-unsubscribe-***@public.gmane.org

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

Scott McIntyre

2002-02-13 03:39:41 UTC

Permalink

Is it grinding at server startup? That would presumably be from refreshing
the genre hash, which is quick and dirty and expensive at the moment. It
iterates over the whole hash when it does that.

Anyway, DBM scales and it's dead simple to implement if you're already using
hashes. One line of code (tie blah blah blah). It's just that currently
we're using SDBM, which pretty much sucks but is the only DBM included with
ActivePerl. Berkeley DB is much better if you're willing to install it.

Scott

----- Original Message -----
From: "John Ruttenberg" <rutt-ACeXNUrNULtWk0Htik3J/***@public.gmane.org>
To: <slimp3-dev-***@public.gmane.org>
Sent: Tuesday, February 12, 2002 9:29 PM
Subject: Re: [slimp3-dev] mp3tag database not ready for prime time

Hmm.. it's only using 40MB. SDBM is not the best dbm library.
Berekely dbm and Gnu dbm are the best ones - no silly key/value length
restrictions, and no separate .dir and .pag files.

just

writing a text file.
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Sponsored by VeriSign - The Value of Trust
Do you need to encrypt all your online transactions? Find
the perfect solution in this FREE Guide from VeriSign.
http://us.click.yahoo.com/jWSNbC/UdiDAA/yigFAA/rIp0lB/TM
---------------------------------------------------------------------~->

To unsubscribe from this group, send an email to:
slimp3-dev-unsubscribe-***@public.gmane.org

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

John Ruttenberg

2002-02-13 02:29:10 UTC

Permalink

Hmm.. it's only using 40MB. SDBM is not the best dbm library.
Berekely dbm and Gnu dbm are the best ones - no silly key/value length
restrictions, and no separate .dir and .pag files.

Even if it's only using 40MB, it takes it a *long* time to read it in (or
maybe write it out) and things sort of grind to a halt when it does, even
things unrelated to the server.

I don't really understand what the dbm library is buying us here. Perhaps
somebody can explain clearly and slowly for me why this is better than just
writing a text file.

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Sponsored by VeriSign - The Value of Trust
Pinpoint the right security solution for your company - FREE
Guide from industry leader VeriSign gives you all the facts.
http://us.click.yahoo.com/lWSNbC/WdiDAA/yigFAA/rIp0lB/TM
---------------------------------------------------------------------~->

To unsubscribe from this group, send an email to:
slimp3-dev-unsubscribe-***@public.gmane.org

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

Sean Adams

2002-02-13 01:51:27 UTC

Permalink

Hmm.. it's only using 40MB. SDBM is not the best dbm library.
Berekely dbm and Gnu dbm are the best ones - no silly key/value length
restrictions, and no separate .dir and .pag files.

Make sure it's really using up all the space. Do a du -sk on it - some dbm
implementations may leave large holes in the file, although I've never
heard of them leaving 2G worth of holes.

92 mp3tag.db.dir
40928 mp3tag.db.pag
total 41020
-rw-rw-r-- 1 rutt rutt 192512 Feb 12 06:48 mp3tag.db.dir
-rw-rw-r-- 1 rutt rutt 2055814144 Feb 12 09:01 mp3tag.db.pag
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

John Ruttenberg

2002-02-13 01:33:03 UTC

Permalink

Make sure it's really using up all the space. Do a du -sk on it - some dbm
implementations may leave large holes in the file, although I've never
heard of them leaving 2G worth of holes.

[***@monster junk]$ du -sk *
92 mp3tag.db.dir
40928 mp3tag.db.pag
[***@monster junk]$ ls -l
total 41020
-rw-rw-r-- 1 rutt rutt 192512 Feb 12 06:48 mp3tag.db.dir
-rw-rw-r-- 1 rutt rutt 2055814144 Feb 12 09:01 mp3tag.db.pag
[***@monster junk]$

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Sponsored by VeriSign - The Value of Trust
Pinpoint the right security solution for your company - FREE
Guide from industry leader VeriSign gives you all the facts.
http://us.click.yahoo.com/pCuuSA/WdiDAA/yigFAA/rIp0lB/TM
---------------------------------------------------------------------~->

To unsubscribe from this group, send an email to:
slimp3-dev-unsubscribe-***@public.gmane.org

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

Sean Adams

2002-02-13 01:17:13 UTC

Permalink

Make sure it's really using up all the space. Do a du -sk on it - some dbm
implementations may leave large holes in the file, although I've never
heard of them leaving 2G worth of holes.

Um, yeah, that's bad. My database isn't bigger than 10 MB.
You don't have an infinite loop, do you? Does a 'find . -follow' in your
mp3 dir finish? That could cause this problem, but it would also suck up
all your RAM as well.
Can you get it to grow like that while you watch it? I'd be interested in
seeing what the actual file looks like.
Scott
----- Original Message -----
Sent: Tuesday, February 12, 2002 9:13 AM
Subject: [slimp3-dev] mp3tag database not ready for prime time

-rw-rw-r-- 1 rutt rutt 192512 Feb 12 06:48 mp3tag.db.dir
-rw-rw-r-- 1 rutt rutt 2055814144 Feb 12 09:00

mp3tag.db.pag

That's 2GB of .pag file. Of course, once this gets written a lot of stuff
gets bad (like restarting the server of the scan.)
By contrast, I store this same information (and more) in mysql and it

takes

about 8MB.
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

John Ruttenberg

2002-02-12 23:45:47 UTC

Permalink

Scott McIntyre:
No infinite link loop. I'll try to reproduce. Perhaps you need a circuit
breaker of some sort.

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Sponsored by VeriSign - The Value of Trust
Pinpoint the right security solution for your company - FREE
Guide from industry leader VeriSign gives you all the facts.
http://us.click.yahoo.com/lWSNbC/WdiDAA/yigFAA/rIp0lB/TM
---------------------------------------------------------------------~->

To unsubscribe from this group, send an email to:
slimp3-dev-unsubscribe-***@public.gmane.org

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

Scott McIntyre

2002-02-12 23:19:34 UTC

Permalink

Hmmm, have I mentioned that SDBM sucks? There's a record length limit of
1k, and it seems you hit it. Most ID3 tags are nowhere near that long, but
I guess you found one that was. Not quite sure how to fix that.

Scott

----- Original Message -----
From: "dean blackketter" <dean-qV/***@public.gmane.org>
To: <slimp3-dev-***@public.gmane.org>
Sent: Tuesday, February 12, 2002 10:12 AM
Subject: Re: [slimp3-dev] mp3tag database not ready for prime time

And when I checked mine this morning after scanning last night, it got this:

sdbm store returned -1, errno 28, key
"/Volumes/10.0.1.201/categories/All Artists/Ludwig van
Beethoven/Complete Beethoven Edition Vol 3 CD4/»Die Ruinen con Athen«
op. 113 - Musik zu August von Kotzebues Festspiel - 8. Chor_ »Heil
unserm König! Heil!« -.mp3" at
/Users/dean/slimp3/server/lib/CPAN/MLDBM.pm line 161.

Oh well. Turning it off.

-dean

Egads! That's not good.
Good thing its turned off by default.

-rw-rw-r-- 1 rutt rutt 192512 Feb 12 06:48

mp3tag.db.dir

-rw-rw-r-- 1 rutt rutt 2055814144 Feb 12 09:00

mp3tag.db.pag

That's 2GB of .pag file. Of course, once this gets written a lot of stuff
gets bad (like restarting the server of the scan.)
By contrast, I store this same information (and more) in mysql and it

takes

about 8MB.
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

To unsubscribe from this group, send an email to:
slimp3-dev-unsubscribe-***@public.gmane.org

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Sponsored by VeriSign - The Value of Trust
Pinpoint the right security solution for your company - FREE
Guide from industry leader VeriSign gives you all the facts.
http://us.click.yahoo.com/pCuuSA/WdiDAA/yigFAA/rIp0lB/TM
---------------------------------------------------------------------~->

To unsubscribe from this group, send an email to:
slimp3-dev-unsubscribe-***@public.gmane.org

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

Scott McIntyre

2002-02-12 23:15:34 UTC

Permalink

Um, yeah, that's bad. My database isn't bigger than 10 MB.

You don't have an infinite loop, do you? Does a 'find . -follow' in your
mp3 dir finish? That could cause this problem, but it would also suck up
all your RAM as well.

Can you get it to grow like that while you watch it? I'd be interested in
seeing what the actual file looks like.

Scott

----- Original Message -----
From: "John Ruttenberg" <rutt-ACeXNUrNULtWk0Htik3J/***@public.gmane.org>
To: <slimp3-dev-***@public.gmane.org>
Sent: Tuesday, February 12, 2002 9:13 AM
Subject: [slimp3-dev] mp3tag database not ready for prime time

-rw-rw-r-- 1 rutt rutt 192512 Feb 12 06:48 mp3tag.db.dir
-rw-rw-r-- 1 rutt rutt 2055814144 Feb 12 09:00

mp3tag.db.pag

That's 2GB of .pag file. Of course, once this gets written a lot of stuff
gets bad (like restarting the server of the scan.)
By contrast, I store this same information (and more) in mysql and it

takes

about 8MB.
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

dean blackketter

2002-02-12 15:12:41 UTC

Permalink

And when I checked mine this morning after scanning last night, it got this:

sdbm store returned -1, errno 28, key
"/Volumes/10.0.1.201/categories/All Artists/Ludwig van
Beethoven/Complete Beethoven Edition Vol 3 CD4/»Die Ruinen con Athen«
op. 113 - Musik zu August von Kotzebues Festspiel - 8. Chor_ »Heil
unserm König! Heil!« -.mp3" at
/Users/dean/slimp3/server/lib/CPAN/MLDBM.pm line 161.

Oh well. Turning it off.

-dean

Egads! That's not good.
Good thing its turned off by default.

-rw-rw-r-- 1 rutt rutt 192512 Feb 12 06:48 mp3tag.db.dir
-rw-rw-r-- 1 rutt rutt 2055814144 Feb 12 09:00 mp3tag.db.pag
That's 2GB of .pag file. Of course, once this gets written a lot of stuff
gets bad (like restarting the server of the scan.)
By contrast, I store this same information (and more) in mysql and it takes
about 8MB.
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Sponsored by VeriSign - The Value of Trust
Secure all your Web servers now - with a proven 5-part
strategy. The FREE Server Security Guide shows you how.
http://us.click.yahoo.com/iWSNbC/VdiDAA/yigFAA/rIp0lB/TM
---------------------------------------------------------------------~->

To unsubscribe from this group, send an email to:
slimp3-dev-unsubscribe-***@public.gmane.org

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

dean blackketter

2002-02-12 15:06:57 UTC

Permalink

Egads! That's not good.

Good thing its turned off by default.

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Sponsored by VeriSign - The Value of Trust
Do you need to encrypt all your online transactions? Find
the perfect solution in this FREE Guide from VeriSign.
http://us.click.yahoo.com/vCuuSA/UdiDAA/yigFAA/rIp0lB/TM
---------------------------------------------------------------------~->

To unsubscribe from this group, send an email to:
slimp3-dev-unsubscribe-***@public.gmane.org

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

dean blackketter

2003-02-22 12:59:25 UTC

Permalink

<!doctype html public "-//W3C//DTD W3 HTML//EN">
<head><style type="text/css"></style><title>Re: [slimp3-dev] mp3tag database not ready for
prime t</title></head><body>
<div>And when I checked mine this morning after scanning last night,
it got this:</div>
<div><br></div>
<div>sdbm store returned -1, errno 28, key
"/Volumes/10.0.1.201/categories/All Artists/Ludwig van
Beethoven/Complete Beethoven Edition Vol 3 CD4/ÈDie Ruinen con
AthenÇ op. 113 - Musik zu August von Kotzebues Festspiel - 8. Chor_
ÈHeil unserm Knig! Heil!Ç -.mp3" at
/Users/dean/slimp3/server/lib/CPAN/MLDBM.pm line 161.<br>
</div>
<div>Oh well.  Turning it off.</div>
<div><br></div>
<div>-dean</div>
<div><br></div>
<blockquote type="cite" cite>Egads!  That's not good.<br>
<br>
Good thing its turned off by default.<br>
<br>
>I tried turning on the mp3 tag database and ended up with:<br>
><br>
>     [***@monster SliMP3_Server_V]$ ls -l
mp3tag.db.dir<br>
>     -rw-rw-r--    1
rutt     rutt      
192512 Feb 12 06:48 mp3tag.db.dir<br>
>     [***@monster SliMP3_Server_V]$ ls -l
mp3tag.db.pag<br>
>     -rw-rw-r--    1
rutt     rutt     2055814144
Feb 12 09:00 mp3tag.db.pag<br>
><br>
>That's 2GB of .pag file.  Of course, once this gets written a
lot of stuff<br>
>gets bad (like restarting the server of the scan.)<br>
><br>
>By contrast, I store this same information (and more) in mysql and
it takes<br>
>about 8MB.<br>
><br>
><br>
>To unsubscribe from this group, send an email to:<br>
>slimp3-dev-unsubscribe-***@public.gmane.org<br>
><br>
><br>
><br>
>Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/<br>
<br>
<br>
------------------------ Yahoo! Groups Sponsor
---------------------~--><br>
Sponsored by VeriSign - The Value of Trust<br>
Do you need to encrypt all your online transactions? Find<br>
the perfect solution in this FREE Guide from VeriSign.<br>
http://us.click.yahoo.com/vCuuSA/UdiDAA/yigFAA/rIp0lB/TM<br>
---------------------------------------------------------------------<span </span>~-><br>
<br>
To unsubscribe from this group, send an email to:<br>
slimp3-dev-unsubscribe-***@public.gmane.org<br>
<br>
 <br>
<br>
Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/ </blockquote>
<div><br></div>
</body>
</html>

John Ruttenberg

2002-02-12 14:13:21 UTC

Permalink

I tried turning on the mp3 tag database and ended up with:

[***@monster SliMP3_Server_V]$ ls -l mp3tag.db.dir
-rw-rw-r-- 1 rutt rutt 192512 Feb 12 06:48 mp3tag.db.dir
[***@monster SliMP3_Server_V]$ ls -l mp3tag.db.pag
-rw-rw-r-- 1 rutt rutt 2055814144 Feb 12 09:00 mp3tag.db.pag

That's 2GB of .pag file. Of course, once this gets written a lot of stuff
gets bad (like restarting the server of the scan.)

By contrast, I store this same information (and more) in mysql and it takes
about 8MB.

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Sponsored by VeriSign - The Value of Trust
Do you need to encrypt all your online transactions? Find
the perfect solution in this FREE Guide from VeriSign.
http://us.click.yahoo.com/jWSNbC/UdiDAA/yigFAA/rIp0lB/TM
---------------------------------------------------------------------~->

To unsubscribe from this group, send an email to:
slimp3-dev-unsubscribe-***@public.gmane.org

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

dean blackketter

2003-02-22 12:59:26 UTC

Permalink

<!doctype html public "-//W3C//DTD W3 HTML//EN">
<head><style type="text/css"></style><title>Re: [slimp3-dev] mp3tag database not ready for
prime t</title></head><body>
<div>Egads!  That's not good.</div>
<div><br></div>
<div>Good thing its turned off by default.</div>
<div><br></div>
<blockquote type="cite" cite>I tried turning on the mp3 tag database
and ended up with:</blockquote>
<blockquote type="cite" cite><br>
    [***@monster SliMP3_Server_V]$ ls -l
mp3tag.db.dir<br>
    -rw-rw-r--    1
rutt     rutt      
192512 Feb 12 06:48 mp3tag.db.dir<br>
    [***@monster SliMP3_Server_V]$ ls -l
mp3tag.db.pag<br>
    -rw-rw-r--    1
rutt     rutt     2055814144
Feb 12 09:00 mp3tag.db.pag</blockquote>
<blockquote type="cite" cite><br></blockquote>
<blockquote type="cite" cite>That's 2GB of .pag file.  Of course,
once this gets written a lot of stuff</blockquote>
<blockquote type="cite" cite>gets bad (like restarting the server of
the scan.)</blockquote>
<blockquote type="cite" cite><br>
By contrast, I store this same information (and more) in mysql and it
takes<br>
about 8MB.</blockquote>
<blockquote type="cite" cite><br>
------------------------ Yahoo! Groups Sponsor
---------------------~--><br>
Sponsored by VeriSign - The Value of Trust<br>
Do you need to encrypt all your online transactions? Find<br>
the perfect solution in this FREE Guide from VeriSign.<br>
http://us.click.yahoo.com/jWSNbC/UdiDAA/yigFAA/rIp0lB/TM<br>
---------------------------------------------------------------------<span </span>~-><br>
<br>
To unsubscribe from this group, send an email to:<br>
slimp3-dev-unsubscribe-***@public.gmane.org</blockquote>
<blockquote type="cite" cite><br></blockquote>
<blockquote type="cite" cite> </blockquote>
<blockquote type="cite" cite><br></blockquote>
<blockquote type="cite" cite>Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/ </blockquote>
<div><br></div>
</body>
</html>