Results 1 to 5 of 5

Thread: Store large amounts of text (logs)

  1. #1
    Join Date
    Mar 2011
    Location
    N 11 19' 0.0012 E 142 15' 0
    Posts
    1,590
    Thanks
    57
    Thanked 99 Times in 97 Posts
    Blog Entries
    4

    Default Store large amounts of text (logs)

    Hey all,

    I'm looking to store large amounts of data, what is the recommended way of doing so efficiently?

    Each entry has three pieces of data: ID, Timestamp, String
    Each String has a maximum size of around 300 chars.
    There are massive amounts of entries (1000/user/day estimate).

    The emphasis is on storing the data efficiently (space-wise).

    Searching isn't really an issue, I just need to be able to recover chunks of the data per user.

    What's the best way to structure this? Database or flatfile?
    Table per user, etc..?
    Posting Tips + FAQ | Issues? Feel free to PM me
    - keyboard1333[at]gmail[dot]com

  2. #2
    Join Date
    Jan 2007
    Location
    Davenport, Iowa
    Posts
    1,875
    Thanks
    92
    Thanked 97 Times in 95 Posts

    Default

    For efficiency in retrieving data there is no real difference. Where MySQL excels is in sorting and organizing data. With storing the amount of data you are talking about I would use a single MySQL table. The number of rows per table in a MySQL database is based on what you set it as and can be expanded as necessary.

    See:

    https://dev.mysql.com/doc/refman/5.0...ger-types.html
    https://dev.mysql.com/doc/refman/5.0...ize-limit.html

    The biggest problem in table size limits comes from the allowed size of the database. For me I can have nearly as many databases as I want provided they are no more than 1GB each. I have not come close to a drop in the bucket with filling that size. I tend to store audio files and pictures on my website as flat files and store the location to those larger files in my database. This allows me to keep my database small. MySQL can and is designed to store most any data type, but overall size of the database is often where the limits become important. I would find out what the limits are for your database size.
    To choose the lesser of two evils is still to choose evil. My personal site

  3. #3
    Join Date
    Mar 2011
    Location
    N 11 19' 0.0012 E 142 15' 0
    Posts
    1,590
    Thanks
    57
    Thanked 99 Times in 97 Posts
    Blog Entries
    4

    Default

    I'm running off a vps, so database size isn't a limit (unless there's some built in max database size in sql).

    Will the queries grow slower the larger the database is?
    Posting Tips + FAQ | Issues? Feel free to PM me
    - keyboard1333[at]gmail[dot]com

  4. #4
    Join Date
    Jan 2007
    Location
    Davenport, Iowa
    Posts
    1,875
    Thanks
    92
    Thanked 97 Times in 95 Posts

    Default

    It should not be a problem. What sort of data are you working with? For example, if your string data repeats you may want to consider normalizing your data. Also consider how fast forums, even large ones, retrieve information from the database. When you get into sites like facebook and google the database structure is actually quite different. We have a thread discussing how sites like google perform searches, but I am having trouble locating it at the moment.

    I am not sure what types of queries you may be referring to. One way to test out specific queries would be run the query in your mysqladmin assuming that is what you are using. Another option is to create a bit of a php timer that will print out the microtime elapsed after a query is completed.

    http://stackoverflow.com/questions/4...-a-mysql-table
    To choose the lesser of two evils is still to choose evil. My personal site

  5. #5
    Join Date
    Mar 2011
    Location
    N 11 19' 0.0012 E 142 15' 0
    Posts
    1,590
    Thanks
    57
    Thanked 99 Times in 97 Posts
    Blog Entries
    4

    Default

    Thanks james.
    I forgot to mention above that my current setup is running off mongodb.
    If you manage to find that thread please let me know!

    I did a bit of research and large mongodb databases can be sharded pretty easily to allow distributed storage.
    I'll keep looking into this and run some bench tests.
    Thanks
    Posting Tips + FAQ | Issues? Feel free to PM me
    - keyboard1333[at]gmail[dot]com

Similar Threads

  1. Replies: 2
    Last Post: 10-23-2013, 02:36 PM
  2. Resolved php parsing/reading large text files
    By regicidedelferoz in forum PHP
    Replies: 22
    Last Post: 07-01-2011, 03:24 AM
  3. Replies: 8
    Last Post: 11-05-2009, 06:09 PM
  4. Codes for user count and payout amounts
    By MLLanser in forum Looking for such a script or service
    Replies: 1
    Last Post: 05-02-2008, 02:23 PM
  5. Importing large text files into MYSQL
    By unixmonk in forum MySQL and other databases
    Replies: 4
    Last Post: 10-15-2007, 07:54 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •