September 27, 2017 - SNAME OpenCalc Status
Steve Hollister is giving a status talk about SNAME OpenCalc on Sept. 28, 2017 for the NY Metropolitan section of SNAME in NYC. A formal paper can be obtained by clicking on the link at the bottom of the "Key Information" section on the right. The talk is expected to be recorded and offered as a SNAME webinar.
For the past two years, OpenCalc has been unfunded, but work is going on apace. It has become clear in the last year that the OpenCalc approach has far-reaching implications that go well beyond NAME, especially in the area of Multi-Disciplinary Optimization (MDO). (See my SNAME paper on the right that I gave last year in Seattle called: "Automatic Hull Variation and Optimization Using SNAME's OpenCalc System".) One cannot automate these complex calculations if they are tied to interactive user interfaces or use proprietary/mixed data file formats.
Since no other programming organization attempts to solve this problem, OpenCalc is being divided into two levels: a base level that offers support for any industry (www.trisplit.com
) and a NAME level built on top and defined by the OpenCalc system. Industries have to take control over their own data definitions in an open way that allows everyone to add calculations and other processing to a common computational toolbox. After 50+ years of software development, it has become clear that we cannot rely on market demand and poorly compatible independent software suites to get the job done, especially in the area of conceptual and preliminary design. It is also clear that NAME engineers need compatible and simple tools to create their own unique computer solutions.
So what started with a call from Rik van Hemmen and turned into Project 114 ("I am not dead yet") and then into OpenCalc and TSPA has evolved into something that can affect how programming is done in all fields. OpenCalc has gone through two working revisions and Rik has shepherded intern and student projects to show it's potential. All that's left is to finish some changes to make the components more general and complete.
January 14, 2017 - What's so important about XML?
When computer programs were first developed, data was read from punched cards (and later on from text files) that contained specific data on specific lines with specific formats. Some of us remember punching cards for the exact Hollerith column requirements of FORTRAN and the US Navy's Ships Hull Characteristics Program (SHCP). Who remembers which columns the 77777s were supposed to be in originally? (was it 72-80?) While computer technology has evolved in extraordinary ways over the last 40+ years and we have eliminated the need for card or file input to control programs, we are still stuck using those old fixed data formats, like SHCP, GF, and OFF, defined by the original programmers. The hydrostatics calc engine I provided to OpenCalc is able to read a variety of those old hull station shape formats, and while it's good to have tools that can read/write legacy data formats, there are new methods that offer insight into how common industry data definition and transfer could be and should be in the future.
Rather than have independent software developers define their own data and file formats in binary, proprietary, and incompatible ways, industries have to take control over their data. This should be more than the ISO/AP/STEP attempts to define "neutral" transfer formats like ISO 10303 for generic CAD data. While they are useful for transferring vast amounts of detailed design data accurately (?) between large scale CAD systems, those formats are not appropriate for conceptual and preliminary design where calculations and tools come from many different developers and vendors. Many know the ugly problems with DXF and IGES transfer of geometry data, and there is nothing available for variable (like LOA and BWL) transfer. Also, neutral file definitions require translators to and from each independent data file definition. With ISO, this often requires buying and learning many different and complex standards and then coding and testing each one. For our small market, that won't happen. It also shouldn't be that complex. When I read an ISO document, my head spins and makes me want to take a nap, and I have probably written over a million lines of code over 40 years and had to learn way too much new technology. (That's the story of my career - lost programs due to changing technology in a small market.)
The world of conceptual and preliminary design is fundamentally different than that of detailed design. It is not a subset of, or a simple matter of scaling down detailed STEP or CAD data. Conceptual design has different levels and types of definition (e.g. station, polygon, mesh, or NURB surface definitions of hulls) and computation needs that have to be provided by many different programs and sources, even individual naval architects. Those who wish to add pieces to "our" data and calculation puzzle shouldn't have to be computer scientists or those employed by competing independent software developers. In many cases, needed calculations are too small to justify commercial development. OpenCalc with separate calc engines and common XML data defintions and open source input/output code is the solution.
What's so different about XML? First, the data definitions or schemas are not defined by each software developer with binary or proprietary data and formats. Our data will be defined by SNAME and one of the goals of OpenCalc is to set up that process and provide free/open source tools for reading and writing that data - not to/from a neutral file format, but directly to/from an industry standard. Programmers will be able to download a program template where all they have to do is to add a new subroutine for a calculation. It can then be compiled into a stand-alone EXE calc engine that can be immediately used by the open source Excel macro-driven front end or combined with other compatible calc engines. I also have a goal of making it automatic to launch a general calc engine from any CAD program.
Second, an XML file is just a stream of characters with data that is grouped by tags, just like HTML. Many already know some tags for HTML (an XML standard), like the one for a paragraph (<p> ... </p>). All data in the file is surrounded by these tags and don't have to be on any particular line or column of a file. In many cases, the ordering of the tag data doesn't matter. However, the most important idea behind XML is that users can add in their own custom tag data. When a program reads an XML file, it will only use those tags it needs and ignore the rest. Gone are the days when data files had ONLY the data required for that one application. You can have a generic XML file that contains all sorts of tag data, like a database, and the application just picks out the data that's needed. That is exactly how my wrapped sequence of calc engines were automated without writing a line of code. They all read the same XML file of design variables and read and wrote only those variables and data structures that were needed. Data doesn't have to be tied to any one file.
The keys to OpenCalc are the separation of the calcs from the user interfaces and the industry-defined and open source input/output code for common XML design variables and geometry data. Gone are the days when data are defined by independent software developers. We need the framework and tools to become creators and not just users.
January 2, 2017 - Web-Based Computing With OpenCalc
The key ingredient of OpenCalc is the separation of calculations from user interfaces. There is (almost always) no need to attach or bond calculations to a custom user interface, making it more difficult to build and test and making the calculations unusable for other purposes. My second paper that I gave in Seattle (see link on this page) shows how separate hull variation, hydrostatics and Holtrop resistance "calc engines" can be combined automatically to create one new (wrapped) calculation that can be launched by the standard Excel front end. These programs can be run separately or combined without writing a line of code. The Excel front end can be customized by the user. These calculations are separate EXE programs that are compiled and tested separately and can be located anywhere on the web ... in theory. However, I've just tested a case where the separate exe calculations are loaded on OneDrive and launched off of the cloud from a separate computer. The data files are local and you never see that the programs could be located anywhere on the web. Users could create their own calc engines and offer the link to anyone else, or it could be a model for CFD, where users might pay a subscription fee for access. How about OpenFoam? The key is the split of calculations into separate tools that can be used for many different purposes by users and located anywhere in the cloud. Much of my work now is on generalizing an XML file organization/schemas that will allow the definition of any variable data or data structure (including geometry), along with open source code for reading and writing that data. One should never have to deal with neutral data files like DXF or IGES on top of having to write the I/O translation code themselves. I'm putting together an open C++ project template with code (other languages will follow) that will read/write any variable from/to a standard XML file of ship/boat data. Users would just have to add in their own subroutine(s) to create a calc engine that will be compatible with all others. It could be immediately run or combined with other calc engines without writing a line of code. This will lead to code that can launch calc engines from any general-purpose CAD program. You will not have to write calculations inside or each CAD program. A user could create their own calc engine (using the template) and have it work immediately with any CAD program.
I've been looking at MDO (Multi-Disciplinary Optimization) and it struck me that this really defines naval architecture: large structures, hydrodynamics, machine design, systems, and so forth. We just have to create separate calculations that can be combined by users to create new solutions. It's too small of a market to be adequately supported by independent software developers who only target the most common calculations. There has to be a structure and framework that allows everyone to add in their own piece of the puzzle. OpenCalc does that. Naval architects have to become creators of new solutions, not just users who have to wait for new versions of other people's software.
December 18, 2016 - The Glue of OpenCalc
The glue that ties OpenCalc together and makes it work is a
common and open source way to define and transfer data using standardized web-compatible
XML text files. Like HTML (now defined as an XML standard by W3C), individual
software developers no longer will be defining industry data in proprietary and
binary files. Those formats lead to the use of neutral or intermediate file
formats, like DXF, IGES, and ISO/STEP, and they lead to delays and errors. That’s
assuming you can even find and pay for the translator in each direction. I once
did a test where I sent a NURB hull definition from my ProSurf3 program to IGES
to SolidWorks to IGES back to ProSurf3 without making any geometry changes. It
came back in a different form that was a proper NURB definition, but almost
impossible to edit. Many CAD users have seen this sort of problem. Rhino gives
very careful directions on how to best transfer IGES NURB surface data to/from
other CAD programs. It doesn’t always work. On top of all of that, transferring
to/from neutral files is a manual process that cannot be automated for
optimization. The standard computer world “app” model of one user interface per
set of calculations and proprietary data file format makes it impossible to
automate a sequence of separate programs in an optimization process. This is the
reason for separating calculations from user interfaces. This is a new innovation
in the computer world that I have defined and it has been first proven with OpenCalc. Wrapping
calculations together has not been done before using separate executable programs
that come from different developers and located anywhere on the web. This can
now be done without writing a line of code or script.
A number of industries are starting to take control over “their”
data definitions so that all parts of their industry can build compatible tools
using a common format. A goal of OpenCalc is to not only define common industry
data structures in an XML file format, but to provide open source code
to read and write those files. Common data definitions eliminate the delays and
incompatible definition problems of transferring data to and from neutral
files, and the open source makes it easy for everyone (not just independent
software developers) to add in their own piece to “our” complex set of design tasks.
The problems with ISO/STEP/AP definitions are that (besides being expensive just to buy the definitions and
mostly for detailed CAD design and not conceptual design), they are complex to understand
and they provide no file Input/Output code. Another goal of OpenCalc is to make sure
that the data organization allows use of legacy hull definitions, like SHCP and
GHS. My 3-program wrapped sequence of HullVary-Hydro-Holtrop uses XML file
transfer for variables, but uses a GHS file definition for station hull data.
Remember that these three calculations are separate EXE files that can be run
separately or combined together in an automated process defined by any user.
The glue is the common data definitions and open source I/O code.
important benefit of XML and its <tag> structure of wrapping data is that
anyone else can add in additional tag data for their own needs. It will not
affect the routines that read and write the files. If a program does not need
or understand a new tag, it will be ignored. You won't necessarily have one file for hull stations and another file for variable data. They can all be mixed in one or more files and wrapped with their XML <tags>. There will not be many file format definitions. There will be just one XML data schema and the tags can be mixed into any combination of files. Data definitions and structures will be separated from file layouts. An XML definition (like HTML) is just a stream of characters with tags that organize the information and a file is just a stream of characters.
My goal with OpenCalc is to make compatible
tools easier to build and combine - to turn engineers back into creators of
solutions instead of just users of apps. Industry-defined data using XML files
allow that to happen, and the XML definitions will be compatible with other
useful XML file definitions, such as HTML and SVG. It will open up our world to
the development of compatible (and perhaps free/open source) tools that will
November 11, 2016
Adding Scalable Vector Graphics (SVG) open source code for output of graphics from any Calc Engine. This one is a section view of a hull midship section from the Hydro Calc Engine. SVG output can be used in many different programs such as Excel and Word.
November 8, 2016
Added three links from Rik van Hemmen in the Key Information section
1. Making Admiral Meyer Smile
2. Project 114 and Student Use
3. Project 114, Research Vessel Optimization
The first link above is a key one. "(build a little, test a little, learn a lot)" Top down systems analysis and development of new methods gets stuck in "analysis paralysis" because one can never know all that's needed ahead of time. I prefer what I call outside-in systems analysis, where you start with a top-down analysis, but quickly start building a working system from the bottom up. Project 114 (now OpenCalc) has gone through two years of bottom-up development and has proven its viability and worth. There is no other way to provide this computer design flexibility and to provide naval architects with tools they can directly combine and control to create new solutions without having to write a line of code.
November 7, 2016
A short new white paper has been added to explain how OpenCalc can and will tie in with general and hull-specific CAD design programs. This allows users to write a separate calc engine that can be immediately used with a hull being interactively defined in any CAD program. This opens up the world of interactive design to those calculations not general and popular enough for development by independent software developers
November 6, 2016
Thank you to all who came to my paper and technical session at SMC2016 in Seattle on Nov. 3rd. Many new changes have been made to Project 114 this year, including a name change to SNAME OpenCalc to reflect the open source structure of the project. Again, this does not mean that SNAME is getting into the software development business, but it does mean that SNAME will provide structure and organization for industry code and data definitions, and pick up where independent software developers and market forces leave off. OpenCalc will also provide mechanisms to revive legacy code that has been lost due to changes in technology and put them into a "calc engine" form that can be separately validated and used for many different purposes. The big technological change this year was the move from using Comma Separated Value (CSV) files to a more formal and general eXtensible Markup Language (XML) form. This change provides a user-customizable form for data and allows general calculation engines to share common data in a standard way - all data is stored in one text file that can be used by all calc engines with free source code for all input/output.
OpenCalc is far more than an organization for lost and unavailable calculations. It defines a new way for developing software by breaking the process into three independent parts: Calc Engines (CE), User Interface Frameworks (UIF), and common industry data definition files (XML). It is based on my idea that calculations never have to be programmatically tied to one custom user interface. Calculations can be separated, tested, and validated using any computer language and fixed into a simple executable (EXE) program that reads a set of input from an XML text file and writes the results back to that same text file using common open source code that ensures that all calculations can work together automatically. This was shown in my SMC2016 paper by combining the three separate calc egines (Hull Variation, Hydrostatics, Holtrop resistance) into one single new calc engine that automatically processes the three calculations in a row. This can be done using any set of calc engines by the user without having the source code or writing a line of new code.
Processing of one or more combined calc engines then falls to the responsibility of standard and open UIF frameworks. Independent calc engines (EXE programs) can be launched from any computer program (unseen in the background) so that you can provide any sort of standard user interface, but most calculations have the same needs: do one calculation for a set of input, loop through many values of one input variable, loop through a set of values for two input variables, graphing, contouring, searching, finding minimums and maximums, and more. There is no need to write custom user interfaces for each set of calculations that do these same sorts of things. Last year, Project 114 showed how these calculations can be done for any calc engine using a standard and open Excel spreadsheet with VBA macro code. This one open Excel UIF spreadsheet can automatically run and analyze any separate calc engine without writing a line of code. My paper this year showed how this same Excel front end processing could be done for any "wrapped" set of existing calc engine programs. This is something that cannot be done if the individual calculations are fixed and bonded with one user interface. All design data is transferred between calc engines using a common XML text file "database" of design data. The common XML files (with open/free I/O source code) ensures that the data is properly transferred with no need for neutral file format conversions to/from separate program data file formats. In addition, other classes or UIF frameworks are possible.
Another goal of this project is to provide an open Excel UIF framework spreadsheet that will resolve and analyze any set of static or dynamic forces and moments of a free body diagram where each force and moment can come from one or many independent calc engines. For example, the standard Excel UIF could analyze the airfoil forces produced for any combination of sailing rigs as produced by an independent calc engine. However, this same separate calc engine could also provide specific forces and moments to a UIF framework that analyzes a free body diagram of both rig and hull forces, where the hull forces and moments would come from a separate calc engine. This FBD calculation is what standard sailboat Velocity Prediction Programs (VPPs) do. However, with this standard and open UIF format, any user could add in their own force or moment calc engines without writing a line of new user interface code. In most cases, independent calc engines can be used for many different purposes and they can be combined and launched by open source Excel spreadsheet front ends. This cannot be done if calculations are tied or bonded to one fixed user interface front end. This is a fundamental flaw of the current "app" model of program architecture.
Note that the current download does not contain the latest XML file format changes. This year's changes (download) won't be available for a month or two. More testing and documentation is required. However, for those who wish to get a head start on this new organization, the old download has a lot of details that are still the same. The data definitions and data file transfers are now done using a more general XML file format. See the "Key Information" section on the right for links to historical and current information, including last year's WMTC paper and this year's SMC2016 paper.