Wednesday, March 21, 2012

Base64 Encoding Binary Files in PT 8.52

My blog contains a handful of posts showing various methods for base64 encoding data (both plain text and binary). While these procedures apply universally, my primary motivation was (and is) integration: sending base64 encoded binary data to other systems. Today I was looking through the 8.52 Integration Broker PeopleBooks and noticed a handful of new features to support binary file operations:

What about files uploaded by users? How do you work with uploaded files? Uploaded files are stored as file attachments in either a database record or FTP destination. The File Attachment API provides the GetAttachment and PutAttachment PeopleCode functions to help you move files in and out of file attachment repositories.

As of 8.52, it seems we no longer need to use the Apache Commons Codec Java library to convert binary files into base64 or base64 into binary files. Likewise, prior to 8.52, if someone asked me if PeopleCode could create a binary file, I would say, "No, but you can use the Java API classes in PeopleCode to write binary files." Now I can say, "Yes, but you will first have to convert your binary data to base64." Hmmm... still sounds a bit funny. Is it better? I think so. Reading and writing binary was the hard part. With 8.52, I have that functionality without having to maintain an extra Java library in my class path. Besides pluggable encryption, as I and my readers demonstrated in my post Base64 Encoding for PeopleSoft, there are a number of ways to encode data including using database features and delivered Java methods (for SQL Server, see sachin's comments here and here).

28 comments:

Rbop51 said...

Jim - I have an unrelated question you may be able to answer. What's the best way to manage memory usage when using, app engines which calls an app package, in turn which uses other app packages and CI's? Cancel the CI after each save and then reinstantiate it? Keep the original instance of the CI and repeatedly do new gets/creates (I don't even know if you can do this?). I guess my bigger question is how does memory usage work with these objects and how is it cleaned up? I'm basically running a custom process and it's running out of memory after 10,000 rows. But, I'm cancelling the CI after each row, the original data is in a 20,000 row rowset. Thanks

Jim Marion said...

@Rbop51, that is a good question. You may want to post it to the OTN PeopleSoft forum. From what you are seeing, it does appear that the component buffers are staying in memory and not getting cleaned up. I thought you had to create a new CI instance for each instance. Once you call get or create, I think you are done with that one and have to start over. I could be wrong though.

Perhaps another approach would be to create a Job that has a couple of app engines, and each one uses a different segment of the data? This assumes that you can partition the data and treat it as subsets of data.

Rbop51 said...

Thanks Jim, will do. I haven't fully analysed it yet, but much to my surprise it's not blowing up when I use the same CI instance over and over and do a new get. Doesn't mean that it's working correctly though. I'll keep you posted.

Rbop51 said...

Jim - You are correct it doesn't work if you leave the CI instantiated. I solved my problem by throwing the kitchen sink at it. I added commitwork(), collectgarbage() and cancel after each row and it ran through them all no problem. Also it doubled the speed.

Thanks,
Russ

Jim Marion said...

@Russ, that is very good to know. I appreciate the response. Cutting the run time in 1/2 is very impressive.

How are you handling restart if it fails? Do you mark rows as processed so they won't be queried and reprocessed?

I wonder if you could fine tune it even more by only calling CommitWork and CollectGarbage every 500 or 1,000 rows or so? This would cut down on disk writes and may reduce memory operations, but not sure about that one. Not sure if it is worth the effort to investigate.

Just FYI, I've seen similar issues with Integration Broker and CI's. I suspect the same solution would help with IB memory issues as well.

For other readers of these comments, you can view additional comments on Russ's OTN thread.

Alex said...

What if you are needing to use Base64 dynamically, like as part of a web service? Leveraging some comments from your blog we came up with this, which is testable with the App Designer debugger:

Local string &myEncodedString, &plainTextXMLData, &myDecodedString;
Local JavaObject &EncodeDecode, &StringObj, &ByteObj;
Local boolean &bResult;

&plainTextXMLData = "Input sample Text.";

&EncodeDecode = CreateJavaObject("com.peoplesoft.tools.util.Base64");

/*Encode*/
&ByteObj = CreateJavaObject("java.lang.String", &plainTextXMLData).getBytes();
&myEncodedString = &EncodeDecode.encode(&ByteObj);

/* Decode*/
&StringObj = CreateJavaObject("java.lang.String", &EncodeDecode.decode(&myEncodedString));
&myDecodedString = &StringObj.toString();

Jim Marion said...

@doubleDog, right, if you are working with non-file data, for example, strings, then, yes, you will still need to use one of the approaches described in my other blog posts. Your code is a good example. Besides the undocumented com.peoplesoft.tools.util.Base64 method, you may also want to look into the documented Pluggable Encryption method demonstrated in this blog post. The pluggable encryption route requires a bit more configuration, though.

Sri said...

HI, Can any one provide me inputs on the base64 encoding in peoplesoft. Our users upload the files online to FTP , we are encoding these files and sending to Filenet system.
Currently we are at 8.49 tools, we are going to move to 8.52 tools soon.
Will there be any difference for encoding for these versions.
Thanks.

Jim Marion said...

@Sri, PT 8.52 has some new methods that make Base64 encoding easier, but the base64 encoding mechanisms used for 8.49 will continue to work in 8.52. For releases prior to 8.52, I recommend this approach which uses Java Objects to read binary files and convert them to base64 text. Once converted, you can insert them into XML messages. My Base64 Encoding Binary Files post uses the apache commons Base64InputStream because it is easier, but if you can't use 3rd party libraries, you can still accomplish this using delivered base64 encoding libraries. See the comments in this post for details.

mymithraa said...

Jim,

We got one scenario. In one of the integration with third party using WSDL we are getting response with attachment. Below is the raw XML response (Using SOAPUI). We are not sure how to handle this RESPONSE in PS. Please share ur thoughts.


HTTP/1.1 200 OK
Date: Mon, 08 Oct 2012 06:46:54 GMT
Server: Apache
Cache-Control: no-store
Transfer-Encoding: chunked
Content-Type: multipart/related; type="text/xml"; boundary="uuid:b4ee8c4f-7749-4146-a0c8-42ac06e19046"

--uuid:b4ee8c4f-7749-4146-a0c8-42ac06e19046
Content-Type: text/xml


--uuid:b4ee8c4f-7749-4146-a0c8-42ac06e19046
Content-Id:
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

"Application Id","Template Id","Country","Job Code"
"241","262","","123"
"241","301","","123"

Jim Marion said...

@mymithraa, the blog comments editor does not allow you to paste in XML directly, so it is difficult to see the issue here. I suggest that you post your question on the OTN forum or in ITToolbox.

John Ney said...

Hey Jim: Here is a cool way to use the 8.52 function GetBase64StringFromBinary(). We create a graphical calendar for students via a java program and the end result is always a jpeg that we display directly on a page. We have used several methods over the past few years, but I found a really cool way to display this without needing an iscript. Here is the gist:
Assuming you have your jpg in a file on the app server:

Local File &imgFile = GetFile(&str_FileName, "R", "", %FilePath_Relative);

DERIVED.HTMLAREA1 = "<... src=""data:image/jpeg;base64," | &imgFile.GetBase64StringFromBinary() | """/>";

That's it! The image is base64 encoded and then the contents pushed directly to the browser.

The issue I had with using an iscript was the image was dynamic and one-time, so if I deleted it from the database before the page loaded, the image was broken.

Regards,
John

Jim Marion said...

@John, thanks! Nicely done! Great use of the File.GetBase64FromBinary method.

Kyle Windsor said...

Jim - I am working on an 8.52 peopletools system and I am using integration broker to communicate with a mobile device. Up until now, all of the development has been with a REST Service. I now need to be able to recieve attachments from the mobile device that will primarily be images.

My question is if it is possible to do this with a REST service using message segments/MTOM-encoded? I have been looking through peoplebooks and there seems to be an indication that it's possible, but REST services require the REST connectors and MTOM-encoding/message segments require either HTTP Connectors or PeopleSoftServices Connectors.

Thanks in advance for any insight you can shed on this subject.


Kyle

Jim Marion said...

@Kyle, is this a web based app or a native app? Are you using Canvas or file uploads or a native app with a REST service? I assume you are using PUT or POST? With Canvas or a native app, I recommend base64 encoding the file to upload (canvas supports this). Then you can place the base64 directly in your text-based payload. I am not sure how to handle this with a file upload, that is why I make the distinction.

MTOM... I actually have no experience with MTOM yet. You might post the same question on the OTN forum

Kyle Windsor said...

Jim - This is a native app and I am using GET/POST to send/receive data from it. Thanks for the suggestion, this is currently what I have in place.

I've been looking into message segments because it appeared to possibly be more efficient but so far it has just appeared to be using everything we aren't.

I will follow up on OTN and see if there are any suggestions there.


Thanks,
Kyle

Unknown said...

Hi Jim,
I'm using WriteBase64StringToBinary to write a file on the App server, then I need to display it to the user with ViewAttachment.
is it possible to use "file://" instead of "ftp://" on Linux/PeopleTools 8.52 to get the file ?
Thanks.

Jim Marion said...

@Stéphane, file:// is for reading local files and is from the perspective of the browser. It won't transfer files between the server and the client. If you write the file into the HTTP path, then you can use http(s)://, but that may not be safe depending on data security. The safest way I know to transfer binary data from the server to the client is through an iScript with %Response.WriteBinary. The problem with this approach is that selecting binary from the database into a variable of type "Any" is the only way I know to get binary data into a variable.

Unknown said...

Jim, In fact I tryed an IScript with this:

Local File &F1;
Local string &base64string;

&F1 = GetFile(&filename, "R", %FilePath_Absolute);
If &F1.IsOpen Then
&base64string = &F1.GetBase64StringFromBinary();
&F1.Close();
End-If;

%Response.Write(&base64string);


but it does not work as the downloaded filesize is 80k instead of 100k ...

Any idea ? I can't file solution in your books :)

Unknown said...

Jim,
regarding the file:// topic, the idea was to use it with the ViewAttachement function, but I can't make it work on UNIX.

/Stéphane.

Jim Marion said...

@Stéphane, your %Response.Write would write the base64 content. You want the binary content. I was referring to this blog post. Notice that I select from a database table, not read from a file. That is because there is no way to get binary content from a file into a variable (there might be, I just haven't found it).

Unknown said...

Hi Jim,
Thanks for your very valuable help.
I fact I was using a record to store a temp file doing:
PutAttachement
ViewAttachement
DeleteAttachement
But it does not work may be due to the lantency with the browser as it gets deleted before it's opened in the browser ... any clue ?

So finaly I tryed your proposal
and I'm doing:
PutAttachement
Then Call an Iscript where I use your method:
%Response.WriteBinary(&data);
and finally DeleteAttachement in the IScript

So for now I'm happy that it works, but I wonder why it does not work with the full Attachement API.

best regards,

/Stéphane.

Asaikarthik said...

Hi Jim,
We wanted to access the SharePoint Portal from PeopleSoft

MS Blog said...
This comment has been removed by the author.
Jim Marion said...

@Mohd, there are a good 1/2 dozen ways to base64 encode data using PeopleTools. This blog post contains links to those other methods. For zip files, a couple of years ago I wrote the post Manipulating Zip Files with PeopleCode. If you are using PeopleTools 8.53, zip files are even easier with the Java ZipFileSystem, but the older method still works.

prasad.poojari said...

Hi Jim,

Hope you're doing great.

I am using ftp to store file attachments on server, however now I need to store Image to Ftp server. I receive this image in binary format from 3rd party java application. Is there a way through which I can convert this binary image data stored in database to a file on some temporary location on the server. This file in temporary location then can be used to attachment functionality which is already existing.

Thanks in advance.

Jim Marion said...

@prasad.poojari, there are a couple of ways. You can use tables as targets for the File Attachment API. If your images are in a record that fits the format, you can use GetAttachment. If not, then you may need to use database support for exporting files (PL/SQL, etc).

Unknown said...

Hi Jim,

I'm working on consuming a webserive where the attachments in 3rd party system are sent in base 64 format in the response message. I receive a successful base 64 response when I make a call to the API from tools like Postman or RESTClient but I'm receiving an empty response when tried from Peoplesoft. Can you please help?

Thanks