Going to the Grid: What the APEX Interactive Grid Means to You and Your End Users

I will be speaking at Kscope 17 in San Antonio, TX on APEX 5.1 Interactive Grid:

Going to the Grid: What the APEX 5.1 Interactive Grid Means for You and Your End Users

Karen Cannell , TH Technology
When: Jun 26, 2017, Monday Session 2 , 11:45 am – 12:45 pm
Room: Cibolo Canyon 5
KScope_Pic
Topic: Application Express – Subtopic: APEX New Release

The long-awaited APEX 5.1 Interactive Grid region is here – but what does it mean to you and your end users? Need some practical guidance on when, why, and how to adopt the interactive grid region type? This session is for those who have seen the introductory demonstrations, are looking to upgrade to APEX 5.1, and need some advice. Does the interactive grid replace interactive reports? No more tabular forms? This session answers practical functional and technical questions raised by this new region type:

• When and why should I upgrade existing regions to interactive grid?
• Is there an upgrade wizard?
• What features will I gain; what features will I lose?
• What about customizations – will they upgrade?
• How does an editable interactive grid compare to my existing tabular form?
• Which features must still be manually written? Do I still need to write all that collection logic?
• How can I customize appearance?
• How can I add dynamic actions?
• Can I extend the interactive grid functionality?
• Will my end users like it? What about data entry users?

This session compares and contrasts interactive grids with the interactive reports and tabular forms we are familiar with. We will pay particular attention to maintaining or replacing features and common customizations in existing applications: checkboxes, 32K limits, cascading select lists, row-level validations, interactions between columns, and navigation between fields. How does the interactive grid region measure up? Should you upgrade all your interactive reports and tabular forms now?

How we do settings of which features are on or off for a particular Grid and for a particular column is now controlled by javascript. Sound scary? It is really not, you just need to learn where to put what piece of code when.  How to access data in the Grid data model?  We will show you that too.

If you cannot attend, don’t worry, I will be posting bits of that presentation here over the next few weeks.  IF you can attend, I will see you there!

Rest up ~ our Kscope 17 schedule is packed, so don’t expect much sleep or downtime.

Read XLXS Files from APEX 5.0 and Higher – One (or Two) Ways

In my work as a consultant I often have the task of getting data from MS Excel spreadsheets into tables in the database.

If this is a one-time load, the process is simple – use either SQL Developer or, easier yet, use the Data Load Workshop in APEX:  SQL Workshop –> Utilities –> Data Workshop.

If the requirement is for a repeated load or many files, I need to set up some automated process for reading in the data.  I do not want to ask my users to open the XLSX files, save the data to CSV, then upload the CSV files.  That makes for unhappy users, at best.

I am sure there are several options for load XSLX files into Oracle data tables.  I am going to focus on a two I have used (and continue to use) successfully across several clients, across many projects.  And another option that used to be a very good option, that goes away with APEX 5.0 and higher.

Let’s do Bad News First.

The Option That Goes Away:

With APEX Listener, now Oracle REST Data Services, ORDS, we gained a wonderful utility for loading XLS or XLSX files into the database.

I will not elaborate on that here, for reasons which will be obvious in a bit.  I did cover that option, and several other Data Load options, in this presentation, How Do I Load Data, Let Me Count the Ways..  Most of these options still apply.  Yes, it is a PowerPoint, but gives you enough detail to get going.

This ORDS-based utility works fabulous for APEX versions less than APEX 5.0.  It loaded data into a collection. You can process your data however you wish from there.  However, as of APEX 5.0, APEX uses JSON to move data, and the ORDS Excel file upload function no longer works.  Joel Kallman explains this clearly in his blog, Let’s Wreck This Together, here.  It will still work for APEX versions < 5.0 – that is, APEX 4.2, APEX 4.1.  But beware, if you are using this Excel-Upload option of ORDS in APEX 4.1 or APEX 4.2 applications, you will need to make some changes when you upgrade those applications to APEX 5.0 and above.

The XLSX Option That Still Works – READ_XLSX

Thanks to Anton Scheffer from AMIS, we have a PL/SQL package that reads .XSLX files from a file in a file server folder that has been mapped to an Oracle directory.  That sounds like a mouthful, but it is simple.

You have files in a folder.  You or your system administrators arrange for that file system folder to be available to the database – usually a mapping of the physical folder to a virtual folder on the database file server.  Then your DBA creates an Oracle directory that points to the folder, physical or virtual, on the database file system. And grants appropriate grants so you can see and read and/or write from /to that Oracle directory.

Now Anton’s package, READ_XLSX comes in.  download the package, read Anton’s blog post.  I am only going to summarize here.

READ_XLSX consists of two main functions, one to read the contents of an XLSX file into a BLOB (named file2blob), and another to read the contents of that BLOB into records (this one is named “read“).  Cast the results of the read function as a TABLE, and voila, you have the results of your XLSX file.

This statement is the guts of it:

SELECT * FROM TABLE( as_read_xlsx.read( as_read_xlsx.file2blob('MY_ORA_DIRECTORY',
 'MyExcelWorkbook.xlsx' ) ) );

Now if you are like me, you do not want ot have to keep typing that SELECT statement.  So I turn it into a view:

CREATE VIEW MY_SPREADSHEET_V AS SELECT * FROM TABLE( as_read_xlsx.read( as_read_xlsx.file2blob('MY_ORA_DIRECTORY',
 'MyExcelWorkbook.xlsx' ) ) );

Now I can query from the view the same as I query from any other table or view I have access.  And incorporate queries from that view into any packages, procedures and functions that I build to meet the requirements of the task at hand.

When I have a series of files in a folder, I use a version of GET_DIR_LIST to read files from the Oracle directory.  (GET_DIR_LIST is a java utility, wrapped in PL/SQL, that returns the list of file names in a folder in a one-column table). You probably have your own method of doing the same.  Then, in a loop, I use dynamic SQL to point my view to the next file in the loop, call my loading procedure to process the contents of that file, then move on to the next file.  OF course I have all kinds of validations and error checking in there, as much is needed depending on the task at hand.  Once configured, the process is clean, reusable, easily customizable and – best of all – works in APEX 5.0 and above.

I did not use Anton’s READ_XLSX directly.  Why?  At the time I downloaded it (years ago now), Anton’s package handled cells of up to 4000 characters.  The spreadsheets I had to load contains some cells with > 4000 characters.  So I made a slight adjustment to handle CLOBs.  I described those customizations here.  I suspect most persons will not need such a customization.

Another (Better?) Option – EXCEL2COLLECTION Plugin:

Is READ_XLSX the only way? No. Seems like too much work? Yup, it can be.  Anton apparently thought so too, because he came up with EXCEL2COLECTION, a utility APEX Plugin that, with less coding than the READ_XLSX package, will read data from an XLSX, XSL, XML 2003 or CSV file into a PL/SQL collection.  I have also used this plugin successfully across several customers and projects.

I have used the EXCEL2COLLECTION plugin in conjunction with a customized instance of the APEX Data Load Wizard to enable simpler (from a user perspective) upload of XLSX files.  I describe more of that process here.  The APEX Data Load Wizard does not out-of-the-box allow upload of XLSX files – one needs to cut/paste.  Integrating the EXCEL2COLLECTION plugin into a customized Data Load Wizard series of pages makes use of other features of the Data Load Wizard – like the column mapping page – possible.  Beware, that customizing the Data Load Wizard pages and process flow means that your work is subject to break upon any and every APEX upgrade.  This customization of the Data Load Wizard process is unsupported territory.  Note that the EXCEL2COLLECTION plugin itself is supported (by Anton). My use of it in customizing the Data Load Wizard is not.

Bottom line:

Anton’s READ_XLSX remains one of the best methods of reading XLSX data into an Oracle database, when the requirement is to read many XLSX files.  When using APEX, investigate using the EXCEL2COLLECTION plugin; that will make your XLSX load tasks easier.  For one-of data loads, I still use SQL Developer or the APEX SQL Workshop Data Workshop utility.  Need to script the load for Production?  Use SQL Developer in a DEV environment, then use SQL Developer to generate a load script, there are several options for that.

Happy Data Loading!

 

 

 

ODTUG Innovation Award ~ Make Your Nomination Now!

Nominations for the 2017 ODTUG Innovation Award are open now through June 2nd.  Make your nomination now, before your Memorial Day weekend.

We are looking for innovative, exceptional, ahead-of-the-curve applications of Oracle development tools that stand out as examples of “the next big thing”, or maybe the really neat next small thing,  in Oracle technology.   Can’t think of anything? Nah, I don’t believe that.  Consider:

  • Do  you have, or does your co-worker have an unusually creative application or combination of new and emerging Oracle tools?
  • Did you find a new open source project that makes your life easier?  Open source projects are eligible, as long as they integrate some Oracle technology.
  • Does your latest project enable faster, smoother, cleaner adoption of an Oracle tool?
  • Are you envying someone else’s work, saying “Wish I thought of that …”?
  • Have you seen a real neat learning tool that aids others in understanding a new Oracle tool?   All projects, large and small, are eligible.
  • Does your organization have an exception program for attracting or training persons new to Oracle technology?
  • Have you created, seen or adopted a new plugin or feature, large or small, that others may not have seen yet?

We are looking for the stuff that makes you say “Wow, great idea …”  or “Neat stuff …” or “wicked cool …” (you can tell where I am from).  We need your help to get as many innovation examples in before the June 2nd deadline.

Now is the time to make your nomination.

The ODTUG Innovation Award honors excellence in creative, effective, and innovative use of Oracle development tools within ODTUG’s supported communities (ADF, APEX, BI, Database, EPM and Career). The goal is to honor developers – individuals or teams – whose passion and creativity shine through in their application of Oracle technology to address real-world problems.  Note that this year we are not excluding anything or anyone, so projects and individuals from all organizations (Oracle, too!) are welcome.

Note that ODTUG members get to vote this year.  Make your nomination, then vote for it too! Member voting will be combined with judges scoring to determine an overall Innovation Award winner.

Check out the full information on the ODTUG web site here:

ODTUG Innovation Award Nominations

Nominations close June 2nd, so act soon!

ODTUG Innovation Award 2017 ~ Nominations are Open!

Nominations for the 2nd Annual ODTUG Innovation Award are now open!

The ODTUG Innovation Award honors excellence in creative, effective, and innovative use of Oracle development tools within ODTUG’s supported communities (ADF, APEX, BI, Database, EPM and Career). The goal is to honor developers – individuals or teams – whose passion and creativity shine through in their application of Oracle technology to address real-world problems.

We are looking for innovative, exceptional, ahead-of-the-curve applications of Oracle development tools that stand out as examples of “the next big thing” in Oracle technology or as unusually creative applications of new and emerging Oracle tools.  The stuff that makes you say “Wow, great idea …”  or “Neat stuff …” or whatever rad saying you use.

Now is the time to make your nomination.

What is it this year: A new Cloud implementation?  A clever APEX integration? A cross-platform collaboration?  The sky (cloud joke) is not the limit!

New this year is Member Voting – all ODTUG members are invited to vote for your favorite Innovation Award nomination. Yes,  we want YOU to help decide the 2017 Innovation Award winner. Member voting will be combined with judges scoring to determine an overall Innovation Award winner.

Check out the full information on the ODTUG web site here: ODTUG Innovation Award Nominations

Nominations close June 2nd, so act soon!

ODTUG

The 2017 ODTUG Innovation Award nominations will be reviewed by a team of judges according to these general topics:

  • Innovation – The “wow” factor: acknowledges the exceptional and innovative
  • Quality – Not only does it have to be “wow,” but it has to work
  • Applicability – Evaluates subject matter importance, completeness, effectiveness, and accuracy
  • Topic merit – Treatment of a cutting-edge topic, technical complexity, or new and unique approach to a common problem

Judges score will be combined with member voting results to determine the overall winner.

For guidance, a perfect example of innovation with Oracle tools is our 2016 Innovation Award Winner: Mia Urman, AuraPlayer 

AuraPlayer and Oracle joined forces to mobilize the Oracle Forms based collision reporting system for the New York MTA to a new Oracle Mobile Cloud based application without redevelopment. In the past, MTA’s officers had to manually fill out accident reports in the field and then deliver it to a data entry clerk who would then enter the information into MTA’s legacy system.  Using AuraPlayer and Oracle’s Mobile cloud they were able to create a “mobile-ready” version of the application without the need for redevelopment or migration of the underlying Oracle Forms system. To enable this modernization, REST services were automatically generated from the Oracle Forms business processes using AuraPlayer. The REST services communicate with the Oracle Mobile Cloud Service, enabling authentication and push notifications, as well as application monitoring and analytics.

Using this application, MTA employees are now able to gather collision information and in addition, now capture photos of the accidents and documents on their tablets.  This data is automatically updated by AuraPlayer into MTA’s Oracle Forms system.  The mobile app also works offline, even without cell or Wi-Fi coverage, this is especially important when an officer is working in a tunnel or on the middle of a bridge. With the real-time submission of traffic accident reports, MTA can assess damages much quicker, determine if a crew is needed to repair damage, and clear the scene in much shorter time—in some cases minutes, instead of hours. The need to manually type collision reports after the fact was eliminated, and the efficiency, accuracy, and timeliness of the data were improved.

See more about AuraPlayer here.

Congratulations, Mia and AuraPlayer!

 

APEX 5 IR Guts and Performance

APEX 5 Interactive Reports (IR) are powerful out of the box, but one can significantly improve performance by strategic settings of certain key parameters.  The full presentation covers all the options.

When your data volumes are low to medium, there may not be much noticeable difference. But when data volumes are high, or the query is resource intensive, the following advice and settings can have a significant performance impact.

In brief:

  • Tune your query, then fine-tune your SQL query. Any slowness here will be magnified in APEX
  • Check MAXROWCOUNT.   The more rows, the longer it takes APEX to retrieve them.  do users really need ALL the rows? OR is it reasonable to ask users to filter first. (APEX 5.1 Interactive Grid will include features to FORCE users to filter first – nice!) Consider  supplying users who need to Download All a separate pag,e os not all users are handicapped by a high MAXROWCOUNT value.
  • Check Max Rows Per Page
  • Check your Pagination setting.  The X to Y of Z is popular, but the “of Z” part is expensive – APEX needs to retrieve all rows then set up pagination. That All Rows part takes time.  Consider using X to Y, and adding a page item that calculates the Z separately. (THere are various options to do this, a topic for another blog post, ranging from a straight COUNT(*) to a sophistcated use of APEX_IR to get the count of the as-filtered IR query)

These are your big-impact settings.

For more info, see the full presentation via the link below or Slideshare:

As you move to APEX 5 Interactive Grid, these three settings till apply – but you have better pagination options, and pagination works differently – heck, all of Interactive Grid works differently – better!  So watch here for upcoming information on Interactive Grid.

APEX5 IR Guts and Performance

Smart use of the IR settings makes for a better end user experience ~ and that’s the whole point!

Migrate BI to APEX 5?

Just back from presenting Migrating BI to APEX 5 at RMOUG Training Days 2016.  Great conference!  Great experts on all Oracle technology. If you have to pick one short sweet conference a year, this is it.

Ever thought of replacing your under-utilized, expensive or outdated BI tool with a suite of APEX Interactive Reports?  In *SOME* cases this makes sense:

  • Your Data Display requirements are reasonable – i.e. users are happy with seeing and working with a few thousand rows or less at a time.
  • Your BI feature requirements include APEX IR Actions (select, sort, filter, chart, pivot, group by aggregate, save, download), and
  • Your BI feature requirements do NOT include BI analysis, unlimited drill-downs, automatic or integrated analysis (such as percents, rank, etc), drag and drop report creation/analysis, and/or MS-Excel-like features. IF you want these, you have to build them.
  • You have APEX resources to plan, design, build and maintain the APEX app
  • You have Database Architects/Developers to plan, design, build and manage the data structures in the database required to consolidate and serve your data ( materialized views, indexes, or other rollups as needed as licensed for).
  • You are prepared to design, build and tune all for performance (as you always should anyways!)
  • You do NOT require out-of-the-box PDF Printing for the as-displayed data set.  Those who own and extensively use BI Publisher may not want to give up this printing luxury IF moving to APEX means doing so. There are many many printing uses cases for which there are many solutions – ensure you have a solution that fits (and is reasonable to implement) before you jump.

Given all this, it just might work for you  It has for many customers, successfully.  The caveats are:

  • If you need a feature that APEX does not natively supply, you must build it.
  • Dynamic Actions are great -but need to be implemented wisely. They may not always perform well with volumes of data.
    • ex: Dynamic parameter selection => Good.
    • Dynamic parameter selection AND dynamic refresh of the IR with the new parameters (a new query against the source data structure) may/will not perform so nicely, depending on the volume of data.
  • Some features are just not reasonable to build in APEX:
    • ex: True Excel-Like behavior
    •        Dynamic Aggregate adjustments

So – want to replace that aging Discoverer installation?  Consider APEX 5.

In planning pages, menus and features, consider these tips:

  • Plan and meticulously tune all structures – materialized views, CUBEs or ROLLUPS, or whatever works for your data. Anything slow here is magnified slow in APEX.
  • Give users mandatory parameters.  This forces up-front filtering, to reduce the result set to a reasonable size AND gives you a ready-made drill path.  We want reasonable size for reasonable performance of all interactive features.
  • Try 3 to 5 Parameters – More gets tedious for end users if they have to select every time.
    • A Temporal (Year. Quarter, Month), and Spatial ( Country, State, City ) and one or few others specific to your data.
    • Consider using Page Zero or building  a Plugin for displaying the same parameter set on all or several pages. (Yes, Plug-Ins can be used locally!)
  • Use a flag to control when data displays. This allows users to filter (using your parmeters Plugin) first, then wait for data to display.  Plan so users do not incur a big wait right up front.
  • Consider multiple IRs on one page, each containing different aggregates of you data depending on the parameters.  The parameters chosen control which ONE of thes IRs displays at any given time.
  • Consider multiple IRs on one page, each containing different Action Item features.  Authorizations for authenticated users control which ONE of thes IRs displays at any given time.
  • Plan and build drill paths wisely.  Intelligently build useful drill paths.  A drill on every column is not necessary. Drills on key columns are nice to have.

Know Your Users, and know what they really do with the data.  That helps you to design and build truly useful data sets and features.

Still not convinced? On the Fence ?

Consider waiting for APEX 5.1 Interactive Grid.  Previews of this new region type show some promising features that BI folks may find interesting:

  • Drag and drop headings
  • Lazy loading – an option to display the frame then the rest of the data
  • Loading data in pages, as opposed to one big result set.

Doesn’t seem like much, but these are big useability improvements , especially when we are considering paging through volumes of data.  Were I to be starting a BI migration project now,  I would investigate the APEX 5.1 Interactive Grid previews and plan my project accordingly.

In the meantime, see my Migrating tBI to APEX 5 slides here, and  if you have specific questions, reach out:

Aside

 

 

 

Image

RMOUG  – Rocky Mountain Oracle User Group – 2013 is around the corner – check out the RMOUG 2013 Training Days for more information. I’ll be presenting on Tabular Form Terrors and How to Overcome Them – updated APEX tabular form information, with guidance on how to build some of those out-of-the-sandbox features that are not readily built by existing wizards and dynamic actions. See you there!