Jump to content
x=usr(1536)

Directory Service for publicly-accessible #FujiNet sites

Recommended Posts

First and foremost: this is not a feature request.  Featureset identification and feasibility discussion, definitely, but *not* a feature request.

 

I've been kicking around the idea of a Directory Service for #FujiNet users.  The basic idea is that it would allow anyone with a #FujiNet to easily find publicly-accessible #FujiNet servers, as well as providing a way for people looking to run those servers an easy way to publish them as being available.

 

This would be a very lightweight protocol, with some similarities to DNS such as root servers containing a central list of server ip addresses and hostnames.  However, I can also see this including extended records (not totally dissimilar to a DNS TXT record) for each entry that describe things like the system type (physical, emulated via SIO2PC, Linux / Windows daemon, etc.), physical location (either using ccTLDs or FidoNET-style unique identifiers), and other info that may be worth providing (like a 40-character-or-less description of what the server offers).

 

Obviously, this would mean having to support the protocol on the #FujiNet side.  Implementing certificates may also be necessary if secure updates of hostnames, etc. on the root servers are considered necessary (which, IMHO, they should be), as well as UI changes to allow for configuring data sent to the root servers.

 

Again, just kicking the idea around for now.  Perhaps a P2P version of this may be less work, as an existing protocol (BitTorrent?) could be used or adapted as needed.

  • Like 4

Share this post


Link to post
Share on other sites
20 minutes ago, tschak909 said:

COOL! Build something. :) 

-Thom

OK, lemme chew it over :-D

 

I have banged on other folks' network code before, so do have a *little* experience in that department.  Time is the killer, though - right now I'm juggling work, school, and a career change.

 

Gonna think on it some more and see if there isn't something that could be adapted rather than starting from scratch if at all possible, but am open to suggestions 🤠

  • Like 1

Share this post


Link to post
Share on other sites

Thinking about this a bit more, LDAP might actually be a good way of handling it.

 

@tschak909: is there enough beef on the #FujiNet to run an LDAP client, preferably over TLS?

Share this post


Link to post
Share on other sites
42 minutes ago, x=usr(1536) said:

Thinking about this a bit more, LDAP might actually be a good way of handling it.

 

@tschak909: is there enough beef on the #FujiNet to run an LDAP client, preferably over TLS?

Yes.

-Thom

 

  • Like 2

Share this post


Link to post
Share on other sites

Why not use DNS and service records for this? Since @x=usr(1536) plans with some central "distribution point" anyway, this could as well be Thom's fujinet.online DNS zone.

 

This way you only need to do some DNS lookups.

  • Like 1

Share this post


Link to post
Share on other sites
20 minutes ago, DjayBee said:

Why not use DNS and service records for this? Since @x=usr(1536) plans with some central "distribution point" anyway, this could as well be Thom's fujinet.online DNS zone.

 

This way you only need to do some DNS lookups.

I've given this consideration as well (actually, a combination of SRV and TXT records) and agree that it may work.  It's not out of the running, but may not be as flexible as using LDAP.

 

One of the things I'd like to see is the ability to tag a record type as being more than just the equivalent of a DNS A record.  In other words, being able to specify that it's a URL to a remote TNFS server, file type, located in a particular country, etc. within the structure of the entry itself brings up the possibility of having a lot of flexibility on the backend of how to display the data.

 

I'm not certain that DNS can do this as well as LDAP might be able to, given the structure of DNS records.  But I'm definitely not opposed to the idea, and if there's a way for it to be workable then I'm all ears.

 

FWIW, if it starts looking like this may need to be a blend of LDAP and DNS, it may be easier to just spin up an Azure instance and run AD since that's pretty much what AD is at its core.

Share this post


Link to post
Share on other sites
11 hours ago, x=usr(1536) said:

Thinking about this a bit more, LDAP might actually be a good way of handling it.

A quick search and I found this https://github.com/chrisridd/LDAPClient which is for esp32 Arduino. Would need some modification to make it work on PlatformIO.

 

17 minutes ago, x=usr(1536) said:

FWIW, if it starts looking like this may need to be a blend of LDAP and DNS, it may be easier to just spin up an Azure instance and run AD since that's pretty much what AD is at its core.

Whatever you decide to use, make sure it works cross platform. Something like OpenLDAP instead of Active Directory. Maybe they are compatible with each other, I don't know.

  • Like 1

Share this post


Link to post
Share on other sites
15 minutes ago, mozzwald said:

A quick search and I found this https://github.com/chrisridd/LDAPClient which is for esp32 Arduino. Would need some modification to make it work on PlatformIO.

Thanks for that.  Added to the list of things to look into.

15 minutes ago, mozzwald said:

Whatever you decide to use, make sure it works cross platform. Something like OpenLDAP instead of Active Directory. Maybe they are compatible with each other, I don't know.

LDAP can query AD, so that's not a problem.  Either way, anything I may do is going to be done under either macOS or Linux, so someone else would have to pick up the Windows side.

Share this post


Link to post
Share on other sites

Giving this more thought, it may be possible to implement this with minimal changes to the @FujiNet side of things.  tnfsd is where the effort should be concentrated.

 

If tnfsd can be built with the ability to query DNS / LDAP / whatever directory service, it can present that info to the #FujiNet in lieu of disk images, executables, etc.  This should also allow existing search functionality to work.

 

At that point, <insert directory service here> inbound to the root directory server would only be needed to register a given public tnfs instance.

 

That would rule out using the #FujiNet as a tnfs server, at least until tnfsd could be ported to it.  But that's a discussion for another day.

Share this post


Link to post
Share on other sites

OK... It appears as though this might be largely-doable by using the right software stack. I'll start building out a prototype system this weekend, time permitting.

 

Hardware will be a Raspberry Pi 3B+ running whatever current Raspbian (or Raspberry OS, or whatever they're calling it this week) is.  This is likely to move over to Debian under an x64 VM if things go well.  I'll need to do some cleanup on the VMware server to make that happen, which is something I really don't want to get into right now so am going with the RasPi.

 

Software stack:

The basic idea is that OpenLDAP runs on <instance> and provides the LDAP tree describing the public-facing tnfs servers.  FUSE lets that tree be presented as a standard mounted filesystem courtesy of ldapfuse.  Finally, tnfsd points to that LDAP mount for the server data it provides to the public.

 

Getting the four installed is a no-brainer; configuration may be interesting.  ldapfuse hasn't been updated in a long time, and there's no telling how well (if at all) it'll work with a current version of FUSE.

 

First goal is to get tnfsd to present the data to a client.  If that's successful, there needs to be a way for tnfsd to tell the #FujiNet that it's providing server addresses, not executables or media images; this is one change I can see being needed on the #FujiNet side.

 

Second goal is to be able to replicate the LDAP data from the local server to another.  This is to be tested locally first, but with the intention of ultimately making it something that happens across the Internet.  The idea is that this will ultimately allow for geographically-distributed root servers.

 

Final goal is to allow for individual tnfsd instances to register as public-facing with the root servers, thereby putting them in the public directory.  That whole end of things is very much TBD.

 

Any and all assistance with this is appreciated.  I'm perfectly happy to not be flying solo on this if there are better ideas out there :-D

Share this post


Link to post
Share on other sites
1 hour ago, x=usr(1536) said:

OK... It appears as though this might be largely-doable by using the right software stack. I'll start building out a prototype system this weekend, time permitting.

 

Hardware will be a Raspberry Pi 3B+ running whatever current Raspbian (or Raspberry OS, or whatever they're calling it this week) is.  This is likely to move over to Debian under an x64 VM if things go well.  I'll need to do some cleanup on the VMware server to make that happen, which is something I really don't want to get into right now so am going with the RasPi.

 

Software stack:

The basic idea is that OpenLDAP runs on <instance> and provides the LDAP tree describing the public-facing tnfs servers.  FUSE lets that tree be presented as a standard mounted filesystem courtesy of ldapfuse.  Finally, tnfsd points to that LDAP mount for the server data it provides to the public.

 

Getting the four installed is a no-brainer; configuration may be interesting.  ldapfuse hasn't been updated in a long time, and there's no telling how well (if at all) it'll work with a current version of FUSE.

 

First goal is to get tnfsd to present the data to a client.  If that's successful, there needs to be a way for tnfsd to tell the #FujiNet that it's providing server addresses, not executables or media images; this is one change I can see being needed on the #FujiNet side.

 

Second goal is to be able to replicate the LDAP data from the local server to another.  This is to be tested locally first, but with the intention of ultimately making it something that happens across the Internet.  The idea is that this will ultimately allow for geographically-distributed root servers.

 

Final goal is to allow for individual tnfsd instances to register as public-facing with the root servers, thereby putting them in the public directory.  That whole end of things is very much TBD.

 

Any and all assistance with this is appreciated.  I'm perfectly happy to not be flying solo on this if there are better ideas out there :-D

I will try to provide as much support as I can, to answer questions etc. My time is literally being spent on writing and rewriting N:.

 

-Thom

 

  • Like 1

Share this post


Link to post
Share on other sites
26 minutes ago, tschak909 said:

I will try to provide as much support as I can, to answer questions etc. My time is literally being spent on writing and rewriting N:.

Understood, and thanks.  Last thing I want to do is add to the pile either you or @mozzwald already have.

  • Like 1

Share this post


Link to post
Share on other sites

Still banging away at this, but progress is slow due to outside factors.

 

It's also become apparent that a Gopher-type interface with some form of Archie or Veronica for searching would be very useful once multiple TNFS sites are browsable from a directory.

 

No, I am not going to go all 'ooh, shiny!' on that and try to implement it :-D  Someone else can take that one on if they want.

Share this post


Link to post
Share on other sites

Progress has been made.  Here's where things stand at the moment:

 

The tnfs.online domain was registered today in order to give the public-facing servers somewhere to live under a common domain when the time comes.  It'll also make building & testing in an environment closer to what the final one will be easier.

 

The software stack (OpenLDAP / FUSE / ldapfuse / tnfsd) is in place and essentially working at a rudimentary level.  LDAP contains no data at the moment, largely because I'm still working out how to structure it; more on that below.  Barring a pressing need to change these components, I'd expect them to more or less be set in stone.

 

ldapfuse is being weird about mounting the LDAP tree.  It does it, but not in a way that makes it completely navigable, or with permissions that tnfsd is happy with.  This is the first time I've used ldapfuse, so am chalking this up to my unfamiliarity with it; it doesn't behave exactly like other FUSE filesystems do, so there is a learning curve.

 

Re: LDAP data structure: this is completely undecided at this point since I've been focussing on just getting software running.  However, there are certain criteria in terms of remote site capabilities that I'd like to list - for example, if a site allows uploads, set an 'incoming' flag.  Root servers get a 'root' flag, country names are flagged by TLD, etc.  The idea behind this is to make it easy to search by certain criteria, which in turn should make it easy to present the data to the tnfs client.

 

If anyone has ideas, suggestions, or wants to help, I'm 107% in favour and contributions are gratefully received.

Share this post


Link to post
Share on other sites
41 minutes ago, x=usr(1536) said:

Progress has been made.  Here's where things stand at the moment:

 

The tnfs.online domain was registered today in order to give the public-facing servers somewhere to live under a common domain when the time comes.  It'll also make building & testing in an environment closer to what the final one will be easier.

 

The software stack (OpenLDAP / FUSE / ldapfuse / tnfsd) is in place and essentially working at a rudimentary level.  LDAP contains no data at the moment, largely because I'm still working out how to structure it; more on that below.  Barring a pressing need to change these components, I'd expect them to more or less be set in stone.

 

ldapfuse is being weird about mounting the LDAP tree.  It does it, but not in a way that makes it completely navigable, or with permissions that tnfsd is happy with.  This is the first time I've used ldapfuse, so am chalking this up to my unfamiliarity with it; it doesn't behave exactly like other FUSE filesystems do, so there is a learning curve.

 

Re: LDAP data structure: this is completely undecided at this point since I've been focussing on just getting software running.  However, there are certain criteria in terms of remote site capabilities that I'd like to list - for example, if a site allows uploads, set an 'incoming' flag.  Root servers get a 'root' flag, country names are flagged by TLD, etc.  The idea behind this is to make it easy to search by certain criteria, which in turn should make it easy to present the data to the tnfs client.

 

If anyone has ideas, suggestions, or wants to help, I'm 107% in favour and contributions are gratefully received.

Need to find (or make) a C API for LDAP. openldap's libraries are a bit big to squooze onto an ESP32.

 

-Thom

 

  • Like 1

Share this post


Link to post
Share on other sites
Posted (edited)
28 minutes ago, tschak909 said:

Need to find (or make) a C API for LDAP. openldap's libraries are a bit big to squooze onto an ESP32.

 

-Thom

 

You may have just obliquely raised a UX point I've been wondering about for a while :-D

 

It seems to me that there are more than a couple of ways to present the server list to @FujiNet clients.  The idea that has been stuck in my head is that by default (meaning: as shipped when a #FujiNet is purchased) there's a 'master' entry for finding TNFS servers in one of the host slots.  Something like 'welcome.to.[fujinet || tnfs || etc.].online' or similar, which is doing a round-robin of the tnfs.online directory servers.  Selecting that gives the highest-level drill-down options, from which the user can get to the server they want.

 

Doing this would hopefully eliminate the need for building an ESP32 LDAP client, since everything is being presented over TNFS.  There may be tweaks needed to the TNFS client on the #FujiNet in order to process certain LDAP attributes presented via TNFS on its end, but I am aiming for as much as possible to be handled server-side in order to keep the changes on the client end as minimal as possible.

 

That approach does remove the possibility of every #FujiNet being able to also act as a TNFS server.  Not ideal, but there are also arguments in favour of keeping that functionality separate.  Totally separate discussion, though.

 

Obviously, it's all up in the air right now, so everyone's mileage may vary ;)  That's where my thoughts are at the moment, however.  Figured it might be helpful if I shared them.

Edited by x=usr(1536)

Share this post


Link to post
Share on other sites
1 minute ago, x=usr(1536) said:

You may have just obliquely raised a UX point I've been wondering about for a while :-D

 

It seems to me that there are more than a couple of ways to present the server list to @FujiNet clients.  The idea that has been stuck in my head is that by default (meaning: as shipped when a #FujiNet is purchased) there's a 'master' entry for finding TNFS servers in one of the host slots.  Something like 'welcome.to.[fujinet || tnfs || etc.].online' or similar, which is doing a round-robin of the tnfs.online directory servers.  Selecting that gives the highest-level drill-down options, from which the user can get to the server they want.

 

Doing this would hopefully eliminate the need for building an ESP32 LDAP client, since everything is being presented over TNFS.  There may be tweaks needed to the TNFS client on the #FujiNet in order to process certain LDAP attributes presented via TNFS on its end, but I am aiming for as much as possible to be handled server-side in order to keep the changes on the client end as minimal as possible.

 

Obviously, it's all up in the air right now, so everyone's mileage may vary ;)  That's where my thoughts are at the moment, though.  Figured it might be helpful if I shared them.

+1 for out of the box thinking. I like this.

-Thom

 

  • Like 1

Share this post


Link to post
Share on other sites
3 minutes ago, tschak909 said:

+1 for out of the box thinking. I like this.

-Thom

 

Thanks :)  ldapfuse really requires the kudos on this one, since it's the glue that could make all of this possible provided I can beat it into submission.

 

Bear in mind that I spent the last 15 years of my now-ended IT career as a manager at one level or another.  I'm nowhere near as technical as I used to be, but I've at least been able to remain a monkey that can occasionally make fire ;)

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...