Content Databases

Here you may share your feedback on any of the online services we provide, or report technical issues that you may have encountered.

Moderator: Staff

User avatar
Roots
Dictator
Posts: 8669
Joined: Wed Jun 16, 2004 12:07 pm
Location: Austin TX
Contact:

Content Databases

Post by Roots » Mon Oct 23, 2006 5:42 pm

Okay, so here's the low-down on what the content databases are and what we need from them.


THE PROBLEM
We have so many art, music, and sound files stored in everyone's user FTP space that we often forget what we already have done.


THE SOLUTION
Write a script that runs on our webserver that periodically snoops through the staff FTP, notes all of the image/sound/music files that it finds, and constructs a series of webpages with these files embedded/linked in them. This will allow us, the staff, to browse through this "content database" and take note on what other people have already done.


ISSUE TO ADDRESS
The staff FTP is very nebulous. That is, people add, rename, move, etc. files in their space all of the time. Everyone organizes their own FTP space in their own way, so we can not do something like rely on the names of directories to tell the difference between a map sprite and a battle background.


REQUIREMENTS
The content database doesn't need any fancy interface or advanced sorting algorithms (yet :devil: ). But I do think that it should support the following organizations for the content that it logs:

- View files by creator (simple: just look at the name of the user in the staff/user/* directory tree)
- View files alphabetically
- View files by type (music, sound, .jpeg, .wav)
- View files ordered by creation date
- View files ordered by last modified date


FREQUENCY OF UPDATE
We'll probably just set this script as a cron job and execute it every few hours, or once every day, or whatever we deem the most appropriate.


FUTURE WORK
Obviously there is a lot of room for growth here, such as being able to keep track of changes to files, or allowing people browsing to "tag" files as sprites, tiles, etc. to spare some of the effort on the part of the script. But lets not worry about those things now, and just get the fundamental design up and running. :approve:


So that's that. Hopefully that will make things much more clear about what it is that we're trying to do with this effort. Please post all questions, comments, suggestions, requests, etc. for this content database here. :hack:
Image
raging_hog
Contributor
Posts: 20
Joined: Mon Oct 09, 2006 2:08 am
Location: Lahti, Finland

Post by raging_hog » Mon Oct 23, 2006 7:18 pm

The walking through files and logging them might be done easily with a shell script(find, grep) but that might hog resources. It also would limit it to *Nix platforms.

The data storage could be XML or DB. DB would of course scale better and XML be easier to port. I don't think there will be many concurrent users for this but the number of files the system logs might be/become quite large. How about DB?

Media files might be best defined by a config file which the script reads before it starts walking through files. What I checked was that there are mostly png, jpeg, ico, bmp, ogg and wav files right now.

The staff members' directories should also be in a config file. (Well duh, I wan't to go to the source every time we get a new staff member. But still, Hard men hardcode hard code)

This kind of a system is heavily depending on the use of filenames. The media filenames now seem to adhere to a standard way of naming so it should be ok(meaning the portrait images are named differently from sprites etc. so we know what media file is what kind of media). If this naming is truly a standard, then the interface could show the files' types(sound fx, portrait etc.).

If we want to extend this system the media file could be coupled with an info file of similar name and some other extension where the creator could comment it(the mood/theme, state of readiness, how good it is etc.) and the system would show that in the interface.

A more hardcore system would let mediaists submit their content and comments with more metadata to a database from which the powers that be could select the media used in game or just view, comment and rate it. Like this(powers-that-be-view):
[img:512:384]http://koti.phnet.fi/hekuma/multimedia_system.PNG[/img]
Of course that would require more work.

The user interface might be done most efficiently with JavaScript so arranging the files by creator, filename etc. wouldn't require a terribly long and boring visit to the web server. Is JavaScript ok? Or should we make a server-side arranging for those fanatic people who simply won't turn their JavaScript on :D? But seriously, maybe some people surf with browsers incapable of JavaScript?

I made some quick sketches of what I understood the interface should look like:

[img:334:172]http://koti.phnet.fi/hekuma/condb_pt1_main.png[/img]
[img:354:175]http://koti.phnet.fi/hekuma/condb_pt1.png[/img]
So from the main view one would click on one of the links which would take the user to the file view(sketch is missing modification date and other such fields). From there the user could download the file or arrange the files again by clicking on "Filename", "Creator" etc.
User avatar
Steu
Developer
Posts: 926
Joined: Thu Jan 05, 2006 3:41 pm
Location: Medicine Hat

Post by Steu » Mon Oct 23, 2006 11:52 pm

The only thing i would change is provide a picture image for all image files
raging_hog
Contributor
Posts: 20
Joined: Mon Oct 09, 2006 2:08 am
Location: Lahti, Finland

Post by raging_hog » Wed Oct 25, 2006 3:38 pm

I've crapped up together a... I mean completed an advanced requirements analysis enhancing prototype of the system which looks practically the same as the sketches I posted. There's a problem with creation times as I'm not sure if they can be reliably acquired from the fs.

The Thing I've now made uses *Nix tools and plain text files with info dependant on newlines so it's hardly portable and probably will be rewritten if in need of lots of expanding(which I believe it is in the future).

The thumbnail feature, which isn't in complete existance yet, might be a bit resource hungry so it might be wise to make it disableable.

Also it would help developing to get ssh access to the host this will be run on and then someone else might test it also. Today I heard I'm getting a job and will start working early next month so I'd like to get this project as far as we can get it before that.
raging_hog
Contributor
Posts: 20
Joined: Mon Oct 09, 2006 2:08 am
Location: Lahti, Finland

Post by raging_hog » Thu Oct 26, 2006 3:12 pm

I went ahead and made the prototype quite functional. Code isn't very pretty now but I'd think that it's possible to refactor it into a more decent version which someone else might also be able to refine if that's needed. It runs through specified(in a config file) path looking for specified(config file) file formats, saves them to file and another component of it parses that file's information and displays it on screen. The rest can be seen from the following media.

Screencaps:
http://koti.phnet.fi/hekuma/roina/hoa_c ... _pres0.png
http://koti.phnet.fi/hekuma/roina/hoa_c ... pres11.png

Flash video:
http://koti.phnet.fi/hekuma/roina/hoa_c ... tation.htm

There's a "No thumbnail"-image now for images that don't have a thumbnail but it might be nice to make it say wav or ogg or whatever need be.

The sorting is written with JavaScript and I don't think it's very scalable now as it takes about second on my machine to sort 16 files but the algorithm could be made a lot faster.
User avatar
gorzuate
Developer
Posts: 2575
Joined: Wed Jun 16, 2004 9:03 pm
Location: Hermosa Beach, CA
Contact:

Post by gorzuate » Thu Oct 26, 2006 7:44 pm

Cool cool cool :cool:

You asked about XML vs DB. Let's go with DB. I think our server has MySQL so that's good.

What do these config files look like that you mentioned?

How feasible would it be to have buttons for say, portraits, tiles, backgrounds, etc. so the user could just click on a button to see which portraits (for example) had already been made?
Image
User avatar
EmreBFG
Team Manager
Posts: 294
Joined: Wed Apr 05, 2006 9:24 pm
Location: Chicago, IL
Contact:

Post by EmreBFG » Thu Oct 26, 2006 10:17 pm

gorzuate wrote:I think our server has MySQL so that's good.


Yep, we certainly do.
User avatar
Rain
Musician
Posts: 1525
Joined: Sun Aug 08, 2004 12:43 am
Location: Granz

Re: Content Databases

Post by Rain » Fri Oct 27, 2006 12:02 am

Roots wrote:THE SOLUTION
Write a script that runs on our webserver that periodically snoops through the staff FTP, notes all of the image/sound/music files that it finds, and constructs a series of webpages with these files embedded/linked in them. This will allow us, the staff, to browse through this "content database" and take note on what other people have already done.




Wow...is that even possible? (Obviously it is or you wouldn't have mentioned it)

I am still amazed that this can be done.
'When Zeon lost his powers, he fell to Earth, and created a giant crator where he hit. His moan destroyed the mountains and the crater was buried by the debris.'

(of Zeon)

Image
User avatar
Jetryl
Artist
Posts: 1485
Joined: Fri Aug 26, 2005 1:35 am
Location: Southern Minnesota, USA

Re: Content Databases

Post by Jetryl » Fri Oct 27, 2006 2:45 am

Rain wrote:Wow...is that even possible? (Obviously it is or you wouldn't have mentioned it)

I am still amazed that this can be done.


I think you're reading too far into it.

It's not classifying the files by some AI voodoo; it's just doing what the Finder always does when it opens a window, and you have it set to "Sort By Kind".

So, it's not going to be able to look at an image, and say "OH, this is a map tile, tee-hee." It's not that smart.
Image
User avatar
Rain
Musician
Posts: 1525
Joined: Sun Aug 08, 2004 12:43 am
Location: Granz

Re: Content Databases

Post by Rain » Fri Oct 27, 2006 3:40 am

Jetryl wrote:I think you're reading too far into it.

It's not classifying the files by some AI voodoo;


In regards to programming, its all voodoo to me Jetryl. :)
Still, I think I understand a bit more clearly what WE can enable the 'finder' to do. Thanks for the explanation.
'When Zeon lost his powers, he fell to Earth, and created a giant crator where he hit. His moan destroyed the mountains and the crater was buried by the debris.'

(of Zeon)

Image
raging_hog
Contributor
Posts: 20
Joined: Mon Oct 09, 2006 2:08 am
Location: Lahti, Finland

Post by raging_hog » Fri Oct 27, 2006 4:42 am

DB would be nice. We can always turn it's data into XML if in need of porting it to somewhere.

Right now the config files look like this:

formats.conf

Code: Select all

png
wav


users.conf

Code: Select all

heimo
luimu


path_to_users.conf

Code: Select all

/var/www/localhost/htdocs/allacrost_test/staff/user


There are three of them because it was simpler to code it that way. Naturally these will be merged into one config file for the final product. Perhaps:

hoa_condb.conf

Code: Select all

PATH_TO_USERS="/var/www/localhost/htdocs/allacrost_test/staff/user"
USERS="heimo luimu"
FORMATS="png wav"


How feasible would it be to have buttons for say, portraits, tiles, backgrounds, etc. so the user could just click on a button to see which portraits (for example) had already been made?


Then we would of course need another component where someone would define what media is needed for the project. Then it would be quite simple to compare the existing media to the needed media to see if something's amiss. And if we make that component then we could also use it to submit media and then mediaists could submit lots of metadata(is it portrait, bg etc.) along with it saving a lot of trouble(see my first post with the image of a HC-system). Of course we could automate that so it would find by itself files from the FTP and add them to the database where someone could then go and fill in the comment fields.

Right now the system depends on the file names so if I go and make a portrait of Claudius with the filename claudieboy.png then the system can't tell if it is a portrait. So the filename must be Claudius_With_Heroic_Pose.portrait.png or something like that.

So, it's not going to be able to look at an image, and say "OH, this is a map tile, tee-hee." It's not that smart.


It might be possible to make an AI to guess if the file is a portrait or map tile etc. from the use of colors in the images(map tiles probably cover the whole image surface while a portrait uses half of it for different colors and the rest is single color). But that's a big might be.

With different browsers there could be problems. I found out about playing sound in the browser and we can make it play sound files directly in the browser but different browsers might handle the sound a bit different and might need a plugin. I installed mplayerplug-in to my Gentoo box with emerge and now Firefox plays the sound like there's no tomorrow. No idea of how other browsers/plug-ins handle it.

Also I think IE and maybe other browsers can't do the JavaScript in the system right now.

So I'd like everyone who wants to use this system to state their browser[s] name and model. And as the JavaScript might have performance problems it would also be nice to know processor speed and maybe RAM.
raging_hog
Contributor
Posts: 20
Joined: Mon Oct 09, 2006 2:08 am
Location: Lahti, Finland

Post by raging_hog » Sat Oct 28, 2006 9:37 am

Audio files can now be previewed via browser now and the main view has now an option to select thumbnail previews. Also there's shown a total number of media file types for the displayed user.

Screencaps updated:
http://koti.phnet.fi/hekuma/roina/hoa_c ... _pres0.png
http://koti.phnet.fi/hekuma/roina/hoa_c ... pres11.png

Code is prettier now although still basically procedural but it should be easy to make OO. Bigger problem is the coupling of the program logic and view as the PHP script just spurts out the HTML instead of using a template engine.
raging_hog
Contributor
Posts: 20
Joined: Mon Oct 09, 2006 2:08 am
Location: Lahti, Finland

Post by raging_hog » Mon Oct 30, 2006 3:05 pm

This thread is starting to resemble a blog more and more. But when the going gets blog the blog gets hogging.

The main menu has some more (working) configuration ability:
http://koti.phnet.fi/hekuma/hoa/conms/s ... _pres0.png

New features:
* Preview images can be forced to retain their own size. This results in:
http://koti.phnet.fi/hekuma/hoa/conms/s ... _bad_1.png

Note the tiny black image in the upper left. The problem of too big previews could be solved by defining that the images retain their own size only if they're smaller than the max width/height. Or maybe we can leave it that way in case there would be a need for big images monitored concurrently. It's the user's decision anyway.

The text fields getting covered by the image should be easy to fix by enhancing HTML/CSS to comply with thumbnail sizes. Too bad I don't like that stuff too much.

* Defining width/height of previews
http://koti.phnet.fi/hekuma/hoa/conms/s ... pres12.png
http://koti.phnet.fi/hekuma/hoa/conms/s ... pres13.png

* Defining the image for file types that don't have a set thumbnail.

* The config file now looks like:
http://koti.phnet.fi/hekuma/hoa/conms/s ... conms.conf
This is for both the component that finds the users' files and the one that displays them.

I was thinking where should the system go to when clicking a file? Let user download it(maybe let user configure a url where all his downloaded files would go like his own ftp site?) or open up a gallery-like view where it's possible to check files one by one? Put in a form where user could mail questions about the file to the creator?
User avatar
Jetryl
Artist
Posts: 1485
Joined: Fri Aug 26, 2005 1:35 am
Location: Southern Minnesota, USA

Post by Jetryl » Tue Oct 31, 2006 2:58 am

raging_hog wrote:But when the going gets blog the blog gets hogging.


I shall quote you on this. :angel:
raging_hog
Contributor
Posts: 20
Joined: Mon Oct 09, 2006 2:08 am
Location: Lahti, Finland

Post by raging_hog » Tue Oct 31, 2006 6:10 pm

Dear diary,

Tonight I installed the preliminary system to the site server:
http://www.allacrost.org/staff/user/rag ... s/main.php

I haven't tested it very thoroughly but it should give an idea of the system and maybe raise some usability issues. The file-walking component isn't in action yet so the file data(date, size) is just something I made up.

Another thing that occurred to me was that should it

A) Walk through users' files and take a snapshot about them, what were they during that time. Then the displayer would show that.

Or

B) Walk just the users' directories and take a snapshot of who the users are. Then the displayer walks the wanted user's files and gathers data about them in display time.

Option A would need it to actually copy all the media to some place. B would be just resource hungry during display time.

Right now it's a mix of both, it gathers data of the files but makes the thumbnail pictures and plays the audio from the real files. There is also a possibility that the real files are deleted by their owner while user is browsing them. And now the users must be added to the config file by hand.

This project is getting more and more interesting, my dear diary, but we shall prevail. Also if anyone wants to design the layout with CSS ask me.
User avatar
gorzuate
Developer
Posts: 2575
Joined: Wed Jun 16, 2004 9:03 pm
Location: Hermosa Beach, CA
Contact:

Post by gorzuate » Tue Oct 31, 2006 9:32 pm

It's good.

Here's what I think about it:

- buttons (file, size, etc.) don't line up with their respective columns
- would it be possible to not thumbnail an image which is smaller than the thumbnail width/height (so it doesn't get stretched when displaying its thumbnail
- I think I like option B best... there's no need to make copies of everything, that could hog up memory real fast. I think it's okay to be resource hungry during display time.
Image
raging_hog
Contributor
Posts: 20
Joined: Mon Oct 09, 2006 2:08 am
Location: Lahti, Finland

Post by raging_hog » Wed Nov 01, 2006 7:13 am

- buttons (file, size, etc.) don't line up with their respective columns


I suck donkey tailsweat with CSS. It should be done by someone more graphical than me and ideally user could define it too(which he/she can do even now using browser extensions or whatever they use).

- would it be possible to not thumbnail an image which is smaller than the thumbnail width/height (so it doesn't get stretched when displaying its thumbnail


I've been thinking about that. Right now the "Force images to use their own size"-option does that but it also shows huge images in their own size. A configuration option could be put in there to force images to use own size just for those which are smaller than the defined width/height.

- I think I like option B best... there's no need to make copies of everything, that could hog up memory real fast. I think it's okay to be resource hungry during display time.


The option B includes walking the user's directories, listing them and making the thumbnails of images. This all means quite bad scalability which might be useful once this project has hundreds of people using this system :) But seriously folks, if a user has hundreds of files the displaying might take long and be heavy for the server. Not to mention if we want to list all users' files it will take a lot more time. I'll do some benchmarks some beautiful day to get a grip on the issue.
User avatar
Roots
Dictator
Posts: 8669
Joined: Wed Jun 16, 2004 12:07 pm
Location: Austin TX
Contact:

Post by Roots » Tue Jan 09, 2007 6:08 pm

Thread necromancy, woot! :D


I've finally looked through the system and the contents of this thread. I think the current implementation is great. We don't need anything fancy or special. Just the ability to automatically see a catalog of all images, sound, and music uploaded on the FTP is more than enough IMHO. Yes, we can speculate about how it would be nice if it had this feature or that feature, but the fact is this is an aid to help us make development of Allacrost easier, and we don't want to spend a ton of time making a wonderful little content management system (that would be nice of course, but I'm here to make a game, not a CMS). :)


If raging_hog thinks its ready for "real use" by us, then I think we should stick it in a special location in the staff FTP and let her rip. :approve:
Image
User avatar
Roots
Dictator
Posts: 8669
Joined: Wed Jun 16, 2004 12:07 pm
Location: Austin TX
Contact:

Post by Roots » Wed Mar 14, 2007 2:36 pm

We need to move the content database to an easier-to-remember URL and start using this. Also, it needs to pick up mp3 and ogg files (it doesn't currently).

Can you give us instructions on how to move it, RH? Is it just a matter of moving the .php file(s)?
Image
User avatar
EmreBFG
Team Manager
Posts: 294
Joined: Wed Apr 05, 2006 9:24 pm
Location: Chicago, IL
Contact:

Post by EmreBFG » Wed Mar 14, 2007 2:56 pm

Roots wrote:We need to move the content database to an easier-to-remember URL and start using this. Also, it needs to pick up mp3 and ogg files (it doesn't currently).

Can you give us instructions on how to move it, RH? Is it just a matter of moving the .php file(s)?
Also, I noticed that the files displayed are not up-to-date; for example, I have more jpegs in my directory than just the pringles can. New members don't show up either, like SimonF. Does the database need to be refreshed periodically?
Post Reply