Documents about protocols like FTP/S, HTTPS/, SFTP and Connect Direct
Mirjana's picture

MFT - SFG - SCC - SCM explanations

Note: This document is created based on Sterling Commerce documentation and articles!!
MFT - Managed File Transfer
IBM® Sterling Managed File Transfer is the market-leading solution for secure data movement that enables enterprises to gain control and oversight of the massive movement of critical corporate data to facilitate data growth, reduce security risk, and improve IT and business efficiency.
The Sterling Managed File Transfer solution is delivered through multiple products that can work standalone, or even better when together. The products are:
  • IBM® Sterling Connect:Direct®—Point-to-point file transfer software optimised for high-volume, assured data delivery of files within and between enterprises.
  • IBM® Sterling Connect:Direct® Secure Plus—An add-on that provides configurable authentication and encryption.
  • IBM® Sterling Control Center—A management solution for all your file transfer activity.
  • IBM® Sterling File Accelerator—High-speed TCP alternative that can increase transfer speed by up to four times.
  • IBM® Sterling File Gateway—An SOA based solution which allows you to incorporate your file transfer communities into all your business processes and incorporates Web-based interfaces for customer self- services and rapid onboarding.
  • IBM®Sterling External Authentication Server - allows you to implement extended authentication and validation services for Sterling Commerce products, referred to as client applications.
  • IBM® Sterling Secure Proxy—An application proxy for securing file transfers across your organisation's demilitarised sone (DMZ).

IBM® Sterling Connect:Direct
Transfer multi-gigabyte files within and among enterprises.
• Near-real-time integration for diverse applications
• Data delivery at the right destination and time
• Proven reliability in demanding industries
• Timely and accurate audit trails and reports for all file transfer activity
• Highly scalable
• Built-in automation and checkpoint restart allows lights-out operation
In this new era of rigorous security and shorter processing windows, IBM® Sterling Connect:Direct is the point-to-point file transfer software optimized for high-volume, secure, assured delivery of files within and among enterprises. Sterling Connect:Direct can deliver your files with:
  • Predictability—Assures delivery via automated scheduling, checkpoint restart, and automatic recovery/retry
  • Security—Ensures that your customer information stays private, and that your file transfers are auditable for regulatory compliance via a proprietary protocol, authorization, and encryption (FIPS 140-2, and Common Criteria certified)
  • Performance—Handles your most demanding loads, from high volumes of small files to multi-gigabyte files
IBM® Sterling File Gateway
Bring B2B file transfer under a service oriented platform.
• Pre-built templates simplify automation for file processes
• Central management of user roles, responsibilities, and policies
with auditing
• Enable business user visibility for B2B document flow
• Comprehensive support for industry-standard protocols
IBM® Sterling File Gateway consolidates disparate centers of file transfer activity, and facilitates the exchange of file-based information securely, in any format, protocol, and file size. With its advanced on-boarding features, extensive communication-channel support, and improved business process management; Sterling File Gateway improves operational execution and time to revenue through a centralized and secure B2B-enabled managed file gateway. Sterling File Gateway solutions offer:
  • A single, secure solution for file transfer—Handles large files and high messaging volumes in any format, any protocol, and any number of external connections
  • Enhanced security and risk management—Offers secure protocols, encryption methods, certificate types, digital signatures, and identity management tools to ensure data integrity
  • Visibility for better management—Leads to better decision making, faster response, and more satisfied customers and business partners
IBM® Sterling File Accelerator™
Speed the delivery of large files over high speed networks.
• High speed transport reduces delays on high capacity networks and dramatically increases transfer speeds
• Built-in congestion control to share available bandwidth with other business applications
• Leverages Sterling Connect:Direct
Digital content explosion is creating bottlenecks
Regardless of the industry, the volume and size of file-based content is causing delays in critical business processes. This extends from check images, patient information, and media and entertainment content, to price updates, backups, and seismic data. Faster network pipes can help, but there is an inherent problem with TCP/IP and network latency that effectively limits how fast TCP/IP transfers can go, regardless of the line speed.
Accelerate your transfers and the business processes they drive
IBM® Sterling File Accelerator diminishes the effect of network latency on large file transfers for more efficient use of your existing large bandwidth line. The result is transfer speeds up to four times faster compared to TCP/IP on the same high speed line. Here's what it means for you:
  • Meet tighter processing windows through a new thinner and stateless protocol that works directly on your existing IP network
  • Cooperatively share the high-speed circuit though built-in congestion control mechanisms
  • Works with IBM® Sterling Connect:Direct® as an alternate transport for our industry leading file transfer product
IBM® Sterling Control Center™
Manage file transfer activity across all your file transfer servers including Connect:Direct, FTP, and Sterling File Gateway.
• Proactive SLA management gives you notification in time to correct problems
• Central visibility and tracking for all data movement activities
• Central configuration management allows you to create,
update, and delete configuration objects with security and logging
• Policy definition and auditing for configurations
As file transfer operations grow, processes need to be monitored and managed across a larger number of servers, applications, business units, time zones, locations, customers, and trading partners. As volumes increase so do exceptions, and your ability to quickly identify, troubleshoot and resolve exceptions will differentiate your business.
Improve customer satisfaction with real-time visibility and control
IBM® Sterling Control Center gives you a consolidated view of your entire file transfer environment—plus the power to respond quickly and efficiently to exceptions, and changes in your environment. Sterling Control Center helps you:
  • Improve SLA performance with centralized exception management, notifications, rules, events, and reporting
  • Meet compliance and regulatory requirements via policy definition, auditing, and reporting
  • Simplify managing your file transfer network through central configuration management 
IBM® Sterling Secure Proxy™
Implement tighter security policies for Internet-based file transfers.
• Session break in the demilitarized zone (DMZ) ensures that no internal service is exposed to attack
• Perimeter security enables the use of the Internet to replace dedicated lines
• Government certified encryption can ensure the safety of consumer data
• Secure protocols, encryption methods, digital signatures, and identity management tools enable you to pass audits
IBM® Sterling Secure Proxy is a demilitarized zone (DMZ)-based application proxy that protects your file transfers from the public Internet, by enforcing tight controls that include trading-partner authorization, multi-factor authentication and session break all before the transfer ever enters your trusted zone. Sterling Secure Proxy will help you:
  • Guard against unauthorized access and reduce data vulnerability to protect your brand
  • Leverage the Internet to lower your file transfer cost and grow your file transfer community
  • Comply with regulatory policies and pass tougher security audits
IBM® Sterling External Authentication Server
Sterling External Authentication Server (EA) allows you to implement extended authentication and validation services for Sterling Commerce products, referred to as client applications. EA includes a server that client applications connect to and a GUI that you use to configure EA requirements.
For SSL or TLS authentication, the connection between EA and the client application is authenticated. Then, the client application sends a request that contains a certificate chain and/or a user ID and password.
  • EA uses the certificate validation or authentication definition that corresponds to the profile name referenced in the request to perform the requested operations.
  • For SSH authentication, the client application sends a request to EA that contains a profile name, user ID, or SSH public key.
  • EA uses the configuration information in the profile to bind to an LDAP directory and look up the SSH key assigned to the user. It also performs an attribute assertion to match the key provided against the list of keys found in the LDAP directory.
  • EA supports a flexible configuration to meet a variety of certificate validation and user authentication and authorization needs. You can configure:
  1. TCP ports (listeners)
  2. SSL/TLS protocol operation
  3. System-wide server connections
  4. Logging operation
  5. Other global system parameters
  6. After you configure the system, create certificate validation and user authentication definitions.
  7. A certificate validation definition specifies validation of certificates against certificate revocation lists (CRLs) and allows validation using attribute queries and assertions. It can include validation using a custom exit to a Java class or an operating system command (for running a program or script).
  8. Authentication definitions configure multifactor authentication using SSL client certificates,SSH keys, user ID and password, and client IP address as factors. They also enable applicationoutputs to allow you to map attributes, such as login credentials that are returned to a query, tooutputs you specify.
IBM® Sterling Connect:Express
IBM® Sterling Connect:Express® is a multi-platform, multi-protocol solution for secure, automated, high performance file transfers over "open" French and European protocols, including PESIT and ETEBAC. It allows for the control, security and automation of file transfers across networks of heterogeneous machines and supports a wide range of operating systems and protocols.
Packaged with its own scheduling capability used for triggering connections to exchange partners Sterling Connect:Express:
  • Manages simultaneous, bi-directional transfers with one or more partners. Partner can have Sterling Connect:Express or any other product that uses one or more of the supported protocols.
  • Optimises flow-through data compression.
  • Integrates file transfers to applications.
  • Offers security functions including system protection and secured access to files.
  • Automates transfers, deliveries, connections and procedures.
  • Supports numerous protocols, equipment and transfer types.
  • Permits the export of large amounts of transfer statistics and information.

AFT - Advanced File Transfer
AFT Advanced File Transfer represented a first generation solution to enable enterprise-level file transfer. It offered consolidated partner configuration and onboarding and enabled streamlined definition of file exchange relationships.
  • Sterling Integrator's Advanced File Transfer (AFT) feature provides reliable, secure, scalable B2B content distribution and Web services across business boundaries, communication modes, and document formats.
  • AFT is a centralized and dynamic file exchange platform for secure transfer of files within and between organizations. It provides end-to-end visibility of file movement in an event-driven, process-oriented, highly scalable framework. These capabilities enable you accelerate new product introduction, improve customer service, rapidly enable AFT partners, and improve operational efficiencies.
  • Sterling Integrator's AFT is built on an extensible Java and J2EE-based architecture that supports comprehensive Internet protocols, document-oriented and stream-oriented processing, advanced application integration, mailboxing, and complete integration with Connect:Direct and Connect:Enterprise UNIX server products. AFT supplies a reliable and secure operational data exchange environment by implementing a policy-based automation and file transfer routing infrastructure.
The primary features of Sterling Integrator AFT are:
1.      Routing – file transfer based on policies and profiles
2.      Visibility – communication adapters record events for monitoring and reporting
3.      Notifications – subscriptions for notification of AFT events to AFT partners by email
4.      Onboarding – streamlines the establishment of AFT partner relationships
5.      Predefined business processes – reduces the number of custom business processes
6.      Extensible – custom features can be added to support additional situations
  • Within Sterling Integrator, you can configure the monitoring capability of the AFT Router. Routing enables a producer of data to direct a file to a particular consumer of that data. In this scenario, the producer and consumer are AFT partners of the router. Partners can be external, such as customers or suppliers, or internal, such as business units of the entity hosting the router.
  • Administrators organize partners into AFT communities for ease of administration and to tailor the set of protocol choices that different AFT partners can employ. Every AFT partner belongs to a defined AFT community.
  • An AFT partner with a mail box accessed over a protocol that is set up by the administrator initiates protocol connections. Alternatively, AFT partners can listen for connections from the router. AFT partners can be either consumers or producers of data. If they initiate connections to their own mail box, they are both a consumer and a producer.
Sterling File Gateway represents the next generation for enterprise-level file transfer. It includes all the features of AFT, and adds the following new capabilities:
  • A Partner still belongs to exactly one community but it can belong to more than one partner group, which is a way to combine partners for business purposes.
  • An Integration Architect can configure the Sterling File Gateway mailbox hierarchy to match that which Partners are already familiar with.
  • The structure for mailboxes is flexibly defined.
  • Sterling File Gateway can perform format unwrapping and wrapping for the ZIP, GZIP and PGP formats.
  • Sterling File Gateway can extract facts from file names and use them for routing and delivery, and as input for generating the file name the consumer sees.
  • Producer and consumer mailboxes are no longer tightly constrained as they were with AFT. Both producer and consumer mailbox patterns can be built from facts available when a routing channel is provisioned; consumer mailbox patterns can also include facts that are only available when a file is being routed.
SFG – Sterling File Gateway
SFG - Sterling File Gateway is an application for transferring files between internal and external partners that can be using different protocols, different file naming conventions, and different file formats.
SFG uses SI as a platform, and every action done through SFG is executed in SI, but web application added for configuration and visibility in SFG is completely new, and one who is uses it does not have to be aware of SI in a background.
SCM – Sterling Community Manager
Sterling Integrator allows you to integrate with Sterling Community Manager to help you manage on-boarding activities with your Trading Partners. You will be able to reduce the need for manual on-boarding and eliminate all duplicate entry efforts by automating the on-boarding process while communicating with your Trading Partners. During the on-boarding process, Trading Partner information is automatically sent to Sterling Integrator to enable a smooth process for transacting with other businesses.
  • When on-boarding a partner in Sterling Community Manager, a series of questions will be asked for both the sponsor and partner that when answered, produces an agreement between the two parties. The agreement represents the information exchanged and agreed upon by both parties in the form of an XML document (the Agreement XML document) or a PDF file.
  • The Agreement XML document consists of all of the pertinent information regarding a single sponsor or partner relationship. This information is processed in Sterling Integrator and referenced from the XpathResponse and XmlTag attributes. The XpathResponse attribute is an identifier that is recognized by Sterling Integrator and allows it to process information from the questionnaire data. The XmlTag attribute is a unique identifier for each questionnaire question block that is used to organize information in the questionnaires.
  • The Agreement XML document processes through a communication channel that securely connects the Sterling Community Manager and Sterling Integrator together. Each time Sterling Community Manager processes trading partner information by way of creation or update, an event is triggered containing the contents of the Agreement XML. An event handler receives the Agreement XML from Sterling Community Manager and transfers the Agreement XML content to the Communication Channel. The Communication Channel consists of a Topic and Queue hosted by a JMS Server. When an Agreement XML is posted to the Communication Channel, it is filtered through the Topic and placed on the Queue for storage until Sterling Integrator is ready to receive the messages.
  • Sterling Integrator provides a JMS Server (ActiveMQ) and has a built-in JMS adapter. The JMS Server and adapter are used to receive agreements from the Queue. When the JMS adapter picks up the agreements, a Sterling Integrator system business process will processes the agreement against a set of converters.
  • The converters parse out the Agreement XMLs and processes them into the Sterling Integrator trading partner data tables. After the application converters are complete, Sterling Integrator will create all the necessary objects to represent the Sterling Community Manager agreement inside of Sterling Integrator. Once Sterling Integrator has processed these records, they are viewable from the Trading Partner > Setup > Advanced > Identities list page (all advanced trading profiles have this option).
Mirjana's picture

SOAP protocol - Web Services

A Web Service is defined by the W3C as "a software system designed to support interoperable machine-to-machine interaction over a network".

Web Services and Web Applications comparison

Web Services are units of code that can be called via HTTP requests in the same way that traditional websites can. The fundamental difference between them is that web services do not normally send HTML in response to a request but instead emit XML. The use of HTTP and XML gives web services advantages of platform and technology independence and allow web services to be set up to work across any intranet or even the internet.

While there are similarities between traditional Web Applications and Web Services the main function of web services is to allow a client to perform remote method calls over the HTTP protocol rather than serve HTML content to a web browser for display. While browser-based Web applications, are loosely coupled and remarkably interoperable, web services are not web based, and as they rely on industry standards, including HTTP, XML, SOAP and WSDL, to expose application functionality on the Internet, they are independent of programming language, platform and device.

While web applications typically are HTML-based, web services are XML-based. Interactive users for B2C (business to consumer) transactions normally access web applications, while web services are employed as building blocks by other web applications for forming B2B (business to business) chains using the so-called SOA model.

Web services use the HTTP transport protocol which is perhaps the most ubiquitous in the world as it is the basis of the internet! Using HTTP means that web services can be hosted within web servers and as such can be available across a company in the same way an intranet sit. But why stop at the company boundary? HTTP is the basis of the internet so why not be available to the world!

This in fact is a key application of web services as it allows Business to Business (B2B) communication

Platform, system and language independence

The HTTP protocol is the main element we have to mention in the story about Web Services, and it is explained above.

The second element of web services is XML. XML in essence is just a well formed text document that can describe data structures using simple tags. Being a text document XML is therefore totally platform and technology independent. Even if you are trying to integrate a .NET application with a Java program, though XML data exchange it becomes easy because both technologies can read text and can understand what is being passed to them from the other program.

So, the 2 key advantages of web services are:

  • Platform/Language independent – Because from the users point of view all that is exchanged is XML which is a text based data format. The machinery that creates this XML behind the scenes can be on any platform using any language. The ubiquity of the HTTP protocol makes it the ideal choice for integration
  • Firewall Friendly – Because they work over HTTP web service traffic travels through port 80 on any PC machine. This means that a System Admin does not have any additional firewall management headaches with extra ports being left open on a machine.

These advantages mean that any machine that has HTTP access to the server and properly technology that can read XML can access the Web Service and access its methods.

Web Service features

Web Services are web-based enterprise applications that:

  • interoperate between different systems
  • interoperate between different software applications
  • can expose any program, functionality, application, class, method, EJB, business object … as a service
  • running on a variety of platforms and/or frameworks
  • simple and extensible
  • use open, XML-based standards and
  • transport protocols (SMTP, HTTP, RMI, JMS, WebSphere MQ …) 

… to exchange data with calling clients running on a variety of platforms and/or frameworks.

Web Services use Extensible Markup Language (XML) messages that follow the Simple Object Access Protocol (SOAP) standard and have been popular with traditional enterprise. In such systems, there is often a machine-readable description of the operations offered by the service written in the Web Services Description Language (WSDL).

  • Web Services effectively eliminate the need for maintaining additional hardware and software
  • Web Services are available anywhere in the world an internet connection is present
  • Web Services eliminate the need to learn other organization's IT Infrastructures
  • Web Services can connect new possibilities to any part of an organization, using any language, or any operating system
  • Web Services can be implemented in hours or days compared to months using traditional implementation methods

WS resume: any program, functionality, application, class, method, EJB, any business object … can be exposed as a Web Service and contacted by WS client independently on system, platform and application (e.g. a Web Service deployed on a J2EE server can be invoked using a C++ client).

Mirjana's picture

Sterling Integrator client components

Protocols such as FTP, SFTP, HTTP, C:D are all divided into suite adapters that can be used to create a client that then connects to the trading partner and transfer the document, enforcing any security specifications from the business process

 Client components for B2B in GIS include FTP, SFTP, HTTP, JMS, MQ, SMTP, POP3, SOAP, C:D, C:E.

 Scenarios in which GIS can act as a client, by using any of above mentioned protocols looks like as follows:

Mirjana's picture

Sterling Integrator server components

Server components for B2B in GIS are FTP, SFTP, HTTP, WS – HTTP over SOAP, C:D Server Adapters

 FTP Server Adapter in GIS/SI

  • FTP Server is tightly integrated with the Application mailbox system. An FTP client can only access the mailbox that is assigned to its user account.
  • This adapter receives and processes requests from external trading partners that are submitted using the FTP protocol. This adapter is used with a Application Perimeter server.
  • Use this adapter to put files into a Application mailbox or get files from a Application mailbox.
  • mailbox activities can trigger routing rules

SFTP Server Adapter in GIS/SI

  • Uses Mailbox subsystem as its repository (virtual roots)
  • Receives and processes requests from external trading partners that are submitted through the SFTP protocol or SCP protocol
  • Use this adapter to enable external SFTP clients or SCP clients to put files into a Application Mailbox or get files from a Application Mailbox.
  • Routing rules for items placed in Mailbox can be used to trigger a business process

CD Server Adapter

  • This adapter receives and processes requests from remote Connect:Direct nodes.
  • Mediate requests originating from a business process and going to an external trading partner using a (non-application) Connect:Direct node or another Connect:Direct Server adapter.
  • Accept requests from a Connect:Direct node (or another Connect:Direct Server adapter) to copy to or from a mailbox or business process
  • Forward requests from one remote Connect:Direct node to another

HTTP Server Adapter

  • Use this adapter to send documents to and receive documents from a trading partner using HTTP
  • Uses Perimeter services
  • Uses the same Jetty HTTP server engine as the GIS ASI console
  • Able to run both WARs and BPML applications
  • Runs application code inside the Application JVM for access to all Application resources
  • Handles SOAP requests as well

Scenarios in which GIS can act as a server, by using any of above mentioned protocols looks like as follows:

Mirjana's picture


The majority of companies exchange essential and non-essential files using FTP (File Transfer Protocol), which has inherent security risks.
The reason FTP has been used in all of these applications is because of its availability -- FTP exists for all operating systems from mainframes down to PCs and it is routinely included in many commercial and open source applications.
Wide availability is a liability
However, this wide availability is also a liability. Because FTP is supported on numerous systems, there is a greater knowledge of the protocol, its operation, and implementations. For example, some companies never change the default passwords of FTP software so their FTP servers are subject to easy manipulation. Careless processes and human error provide an unintentional open door to malicious users who frequently hack into FTP servers and steal data.
In fact, the Sterling Commerce MFT solutions, which Connect:Direct is part of, have never been breached. This ensures information will be protected when a company does business electronically.
Maintaining and patching problem
Since FTP is available on many platforms, many operating systems and with many applications, companies must be particularly vigilant to ensure that security is airtight for all of the impacted servers and applications. They must respond quickly to any newly discovered vulnerabilities and make certain that they are rapidly patched. Since a company may not have control over all of the systems that come into play and might not be able to ensure that all systems meet security mandates, hackers have multiple points of attack to compromise data on scattered FTP servers.
Connect:Direct as a proprietary protocol is not so publicaly known and easy to hack at all.
Encryption and security
Another vulnerability of FTP is that alone it does not provide encryption. Files are sent as is and the content and FTP usernames and passwords are transmitted in clear text, all of which can be intercepted by someone eavesdropping on a communications link.
As security concerns are not a part of the FTP model and a client must supply an ID and password upon opening the connection to the server, this security information is transmitted in open text.
The Connect: products offer multiple choices, ranging from securing FTP traffic to robust security, that allow the data movement operation to fit naturally within enterprise security policies. If support of FTP traffic is required, the data flow can be encrypted. If higher security levels are required, proxy-based security, coupled with authentication and configurable encryption, can be implemented within the Connect: deployment.
The Connect:Direct Secure+ Option for Windows application provides enhanced security for Connect:Direct and is available as a separate component. It uses cryptography to secure data during transmission.
Cryptography provides information security as follows:
  • Authentication verifies that the entity on the other end of a communications link is the intended recipient of a transmission.
  • Non-repudiation provides undeniable proof of origin of transmitted data.
  • Data integrity ensures that information is not altered during transmission.
  • Data confidentiality ensures that data remains private during transmission.
Management, tracking, logging and auditing
Furthermore, as FTP use grows, management, tracking and auditing burdens also grow. Most often, file transfers are part of a larger workflow where completion of a task is predicated on the receipt of a file and then some action being taken on the information in that file. Lacking suitable tracking and auditing tools, an IT manager or corporate executive would be hard pressed to determine whether a transaction was completed or why one failed.
The statistics file stores information about all events that take place within the Connect:Direct server for a specific period of time. Each record within the statistics file consists of fields that contain general information about the record and a field that contains the statistics or audit information to log.
Handling movement workload (Queuing and scheduling capabilities)
FTP provides no way to control critical data movement or balance it against lower-priority work that can impact processing windows. Massive, unmanaged data movement can delay and slow critical deliveries. FTP places all control in the hands of the client, and the first job usually wins. FTP also lacks the ability to create an enforceable policy for workload execution. Over time, this frequently results in chaos.
Connect:Direct gives each process a work-queue priority and a session class. Priorities are used to determine when processes run, and session classes are used to reserve transmission channels for critical transfers. These can be set up and enforced in accordance with business requirements. Users’ requests are always accepted, but the actual operation of the request is scheduled according to the business policy that drives the priority and class structure.
This accomplishes the goals of the user as well as those of the business.
Without queuing, scheduling and management capabilities, it is impossible to control the data movement workload.
As Connect:Direct Processes are submitted, they are placed in one of the four TCQ logical queues: Execution, Wait, Timer, and Hold. As sessions are available, the TCQ releases Processes to begin execution according to the Process class and priority.
Connect:Direct Processes could be scheduled to execute in the future.
In addition to consistent management, organizations need a structured level of notification that enables real-time adjustments to the data movement infrastructure. The enterprise requirements for notification are:
  • Instantaneous notification of critical exception and error conditions
  • Flexibility in the routing of notifications
  • Integration of data movement notification with the Enterprise Systems Management (ESM) structure
  • Historical logging of data movement activities
Connect: answers all these requirements by providing notification and logging as a natural part of the data movement operation. Notification can be routed, using a variety of platform capabilities, to operation and monitoring staff. Alerts represented by SNMP traps can be directed to ESM systems for proactive action at the network level. And all Connect: activity, including finely grained operational detail, is logged continuously.
FTP provides none of these capabilities. It is very difficult, if not impossible, to determine previous FTP activity. Any action that is required must be performed by the client/user. This makes for an inconsistent and unresponsive data movement infrastructure. Therefore, the cost associated with the use of FTP must include the inherent delays in exception discovery.
Connect:Direct for Windows provides two notification methods:
  • NT Broadcast—NT Broadcast notification is performed using the Windows net send command.
  • SMTP—E-Mail notification is performed using Simple Mail Transfer Protocol (SMTP) notification, a simple ASCII protocol.
FTP does not provide an automated way to recover from network errors. Any outage that occurs with FTP operations must first be discovered and then handled manually. This generally means restarting the failed operation from the beginning.
The costs associated with FTP recovery are:
  • Retransmission due to networking resource failure. On average, FTP will need to retransmit half the overall data movement volume per failure. Connect: recovers the network connection and requires no retransmission.
  • During a network-resource failure, FTP use requires discovery of the failure. This delay in restart represents cost. Connect: will automatically sense network failures and retry the operation. In most cases the recovery will be automatic. In the case of a permanent outage, notification is provided to enable rapid resolution.
Several facilities are provided for Connect:Direct process recovery after a system malfunction. The purpose of Process recovery is to resume execution as quickly as possible and to minimize redundant data transmission after a system failure. The following Connect:Direct facilities are available to enable Process recovery:
  • Process step restart
  • Automatic session retry
  • Checkpoint/restart – resume for check point
  • Intelligent session restart -permits a session retry on only the first Process (others ar eplaced in Timer queue) submitted
  • Short-term and long-term retry – restart in specific time periods
Through the use of scripting, scheduling, and application integration, automation can ensure that proven business processes can continue to be successful by eliminating human error. With processes that require human interaction, errors and unintended activity can be introduced into a process flow. Many times such an error will not be discovered until much time has passed.
The cost involved with the lack of FTP automation include:
  • Operations that complete successfully but are not usable since there is no way to validate user selected (or defaulted) options.
  • No central control of scheduled activities. Clients can initiate FTP activity regardless of its importance or impact on schedule.
Connect:Direct File Agent provides monitoring and detection capabilities that enhance the
automation you accomplish with Connect:Direct Processes.
Mirjana's picture


Differences in general:

FTP is file transfer protocol, so it is optimised for transmitting files
HTTP is hypertext transfer protocol, so it is optimised for transmitting hypertext, i.e. web pages.
FTP does a lot more packet verification. That means it's somewhat slower for a given size file, but the file is more likely to get through in one piece because, for a binary application file, for example, one lost packet is as good as losing the whole thing.
HTTP was designed to be fast, but not solid. Who cares if you lose a few packets here and there anyway?
FTP, is a protocol used to upload files from a workstation to a FTP server or download files from a FTP server to a workstation.
HTTP, is a protocol used to transfer files from a Web server onto a browser in order to view a Web page that is on the Internet. Unlike FTP, where entire files are transferred from one device to another and copied into memory, HTTP only transfers the contents of a web page into a browser for viewing
When ftp appears in a URL it means that the user is connecting to a file server and not a Web server
When http appears in a URL it means that the user is connecting to a Web server and not a file server
Statefull protocol and requires fewer round    trips for communication.
Stateless protocol – only request response
FTP is a two-way system as files are transferred back and forth between server and workstation.
HTTP is a one-way system as files are transported only from the server onto the workstation’s browser.
FTP puts less load on the network for smaller transfers
For transfers of large amounts of data, both are comparable
With ftp, clients can do a directory listing and get time stamps and file sizes.
Directory listing is not usually possible with Web connections, because that information is hidden from the user
Can be tied to specific user accounts and require user authentication – PROS
Can allow anonymous access if desired – PROS
Easy target for hackers – CONS
HTTP may be less secure than FTP depending on which software you use for your HTTP server and which platform on which you are running the HTTP server and how you configure it

Differences in details

Both protocols are used for uploads and downloads on the internet, for text and for binary, both over TCP/IP. But there are a lot of differences in the details:
FTP appeared roughly ten years before HTTP was invented. FTP was the one and only protocol back then.
Both protocols offer uploads. FTP has an "append" command
HTTP is more of a "here's data coming now you deal with it" approach.
It could be worth noticing that WebDAV is a protocol on top of HTTP that provides "filesystem-like" abilities
FTP has a notion of file format so it can transfer data as ASCII or binary (and more) where.
FTP thus also allows text conversions when ASCII files are sent between systems of different sorts
HTTP always sends things binary.
HTTP provides meta-data with files, Content-Type, which clients use but FTP has no such thing. The meta data can thus be used by clients to intrepret the contents accordingly
FTP does not send headers.
When sending small files, the headers can be a significant part of the amount of actual data tranffered - CONS
Transfers with HTTP always also include a set of headers that send meta data
HTTP headers contain info about things such as last modified date, character encoding, server name and version and more - PROS
Something related, although not similar, is FTP's support for requesting multiple files to get transferred in parallel using the same control connection. That's of course using new TCP connections for each transfer so it'll get different performance metrics
HTTP supports pipelining. It means that a client can ask for the next transfer already before the previous one has ended, which thus allows multiple documents to get sent without a round-trip delay between the documents, and TCP packets are thus optimized for transfer speed.
FTP involves the client sending commands to which the server responds. A single transfer can involve quite a series of commands. This of course has a negative impact since there's a round-trip delay for each command
Reretriving a single FTP file can easily get up to 10 round-trips - CONS
HTTP transfers are primarily just one request and one response (for each document)
Two connections
One of the biggest hurdles about FTP in real life is its use of two connections. It uses a first primary connection to send control commands on, and when it sends or receives data, it opens a second TCP stream for that purpose - CONS
Uses only one connection for request and response
FTP opens the second connection in an active or passive mode, which basically says which end that initiates it. IT's a client decision to try either way.
There is no active/passive option in HTTP
Firewall and NATs
FTP's use of two connections, where the second one use dynamic port numbers and can go in either direction, gives the firewall admins grief and firewalls really have to "understand" FTP at the application protocol layer to work really well.
This also means that if both parties are behind NATs, you cannot use FTP!
Since firewalls need to understand FTP to be able to open ports for the secondary connection etc, there's a huge problem with encrypted FTP (FTP-SSL or FTPS) since then the control connection is sent encrypted and the firewall(s) cannot interpret the commands that deal with creating the second connection. Also, the FTPS standard took a very long time to "hit it" so there exist a range of hybrid versions out in the wild.
FTP and HTTP have a different set of authentication methods documented. While both protocols offer basically plain-text user and password by default, there are several commonly used authentication methods for HTTP that isn't sending the password as plain text, but there aren't as many (non-kerberos) options available for FTP.
Both protocols offer support for download. Both protocols used to have problems with file sizes larger than 2GB but those are history for modern clients and servers on modern operating systems
Both FTP and HTTP support resumed transfers in both directions, but HTTP supports more advanced byte ranges
FTP must create a new connection for each new data transfer.
For HTTP communication, a client can maintain a single connection to a server and just keep using that for any amount of transfers.
HTTP has a means of sending data which has an unknown size at the start of the transfer, but still not have to use close of the connection to signal the end of the data. This is done by "chunked encoding", which simply sends a stream of [size-of-data][data] blocks over the wire.
Another obvious benefit with chunked encoding compared to plain closing of the connection is the ability to detect premature connection shutdowns.
FTP offers an official "built-in" run length encoding that compresses the amount of data to send, but not by a great deal on ordinary binary data. It has also traditionally been done for FTP using various "hackish" approaches that were never in any FTP spec.
HTTP provides a way for the client and server to negotiate and use (different) compression for the sent data. The gzip algorithm being the perhaps most compact one.
Name based
In FTP, you cannot do name based virtual hosting at all.
Using HTTP, you can easily host many sites on the same server and they are all differentiated by their names.
Dir Listing
One area in which FTP stands out somewhat is that it is a protocol that is directly on file level. It means that FTP has for example commands for listing dir contents of the remote server
However, the FTP spec authors lived in a different age so the commands for listing directory contents (LIST and NLST) don't have a specified output format so it's a pain to write programs to parse the output. Latter specs (RFC3659) have addressed this with new commands, but they are not widely implemented or supported by neither servers nor clients.
HTTP does not have a concept of listing dir content
FTP has always been used over proxies as well, but that was never standardized and was always done in lots of different ad-hoc approaches.
One of the biggest selling points for HTTP over FTP is its support for proxies, already built-in into the protocol from day 1. The support is so successful and well used that lots of other protocols can be sent over HTTP these days just for its ability to go through proxies.
Transfer speed
What makes FTP faster:
No added meta-data in the sent files, just the raw binary
Never chunked encoding "overhead"
What makes HTTP faster:
reusing existing persistent connections make better TCP performance
pipelining makes asking for multiple files from the same server faster
(automatic) compression makes less data get sent
no command/response flow minimizes extra round-trips
Mirjana's picture


SFTP does not have anything common with FTP

SFTP, or secure FTP, is a program that uses SSH to transfer files. Unlike standard FTP, it encrypts both commands and data, preventing passwords and sensitive information from being transmitted in the clear over the network. It is functionally similar to FTP, but because it uses a different protocol, you can't use a standard FTP client to talk to an SFTP server, nor can you connect to an FTP server with a client that supports only SFTP.
SFTP is most often confused with FTPS and vice-versa.  However, unlike FTP and FTPS these protocols are not at all related. SFTP is actually a sub-system of the SSH (Secure Shell) protocol and typically runs on port 22. Unlike FTP/S, SFTP does not have the concept of separate command and data channels. Instead both data and commands are transferred in specially formatted packets via a single connection. Furthermore, unlike FTPS explicit SSL, SFTP encrypts the entire session and does not offer the ability to switch between unencrypted and encrypted mode.

Encryption methods for SFTP and FTPS

Both FTPS and SFTP use a combination of asymmetric algorithm (RSA, DSA), symmetric algorithm (DES/3DES, AES, Twhofish etc.) and a key-exchange algorithm.
For authentication FTPS (to be more precise, SSL/TLS protocol under FTP) uses X.509 certificates, while SFTP (SSH protocol) uses SSH keys.
X.509 certificates include the public key and certain information about the certificate owner. This information lets the other side verify the integrity of the certificate itself and authenticity of the certificate owner. Verification can be done both by computer and to some extent by the human. X.509 certificate has an associated private key, which is usually stored separately from the certificate for security reasons.
SSH key contains only a public key (the associated private key is stored separately). It doesn't contain any information about the owner of the key. Neither it contains information that lets one reliably validate the integrity and authenticity. Some SSH software implementations use X.509 certificates for authentication, but in fact they don't validate the whole certificate chain - only the public key is used (which makes such authentication incomplete and similar to SSH key authentication).

SFTP vs FTPS, what to choose

While FTP is very popular, it has certain disadvantages that make it harder to use. The major drawbacks are:
·         lack of the uniform format for directory listing (this problem has been partially solved by introducing MLST command, but it's not supported by some servers) and
·         presence of the secondary connection (DATA connection).
·         Security in FTP is provided by employing SSL/TLS protocol for channel encryption. The secured version of FTP is called FTPS.
As usually, the answer depends on what your goals and requirements are. In general, SFTP is technologically superior to FTPS. Of course, it's a good idea to implement support for both protocols, but they are different in concepts, in supported commands and in many other things.
It's a good idea to use FTPS when you have a server that needs to be accessed from personal devices (smartphones, PDAs etc.) or from some specific operating systems which have FTP support but don't have SSH / SFTP clients. If you are building a custom security solution, SFTP is probably the better option.
As for the client side, the requirements are defined by the server(s) that you plan to connect to. When connecting to Internet servers, SFTP is more popular because it's supported by Linux and UNIX servers by default.
For private host-to-host transfer you can use both SFTP and FTPS. For FTPS you would need to search for a free FTPS client and server software or purchase a license for commercial one. For SFTP support you can install OpenSSH package, which provides free client and server software.

 SFTP and FTPS – pros and cons

Good standards background which strictly defines most (if not all) aspects of operations
The communication is binary and can't be logged "as is" for human reading
Only one connection (no need for DATA connection)
SSH keys are harder to manage and validate
The connection is always secured
The standards define certain things as optional or recommended, which leads to certain compatibility problems between different software titles from different vendors
The directory listing is uniform and machine-readable
No server-to-server copy and recursive directory removal operations
The protocol includes operations for permission and attribute manipulation, file locking and more functionality
No built-in SSH/SFTP support in VCL and .NET frameworks
Widely known and used
Doesn't have a uniform directory listing format
The communication can be read and understood by the human
Requires a secondary DATA channel, which makes it hard to use behind the firewalls
Provides services for server-to-server file transfer
Doesn't define a standard for file name character sets (encodings)
SSL/TLS has good authentication mechanisms (X.509 certificate features)
Not all FTP servers support SSL/TLS
TP and SSL/TLS support is built into many internet communication frameworks
Doesn't have a standard way to get and change file and directory attributes
Mirjana's picture

Connect Direct protocol

Necessity for secure file movement must ensure data delivery to the right destination within the right time window, so the receiving application can process and act upon it consistently—day after day, and year after year. The problem is many existing file transfer “solutions” built on FTP do not have the necessary mechanisms for management, monitoring, or advanced security.
Connect:Direct is the point-to-point file transfer software optimized for high-volume, secure, assured delivery of files within and among enterprises. It’s the industry-leading solution worldwide for assured, secure delivery of files, especially in the demanding financial services and telecommunications industries.
For those who need added security, Connect:Direct Secure+ is an add-on that provides configurable authentication and encryption
Connect:Direct can deliver your files with:
  • Predictability – assures delivery via automated scheduling, checkpoint restart, and automatic recovery/retry
  • Security – ensures that your customer information stays private, and that your file transfers are auditable for regulatory compliance via a proprietary protocol, authorization, and encryption (FIPS 140-2, and Common Criteria certified)
  • Performance – handles your most demanding loads, from high volumes of small files to multi-gigabyte files
For reliable and secure file transfer, file transfer provided must be near real-time integration needed for applications as diverse as convergence billing in telecommunications, synchronization of central and disaster-recovery sites, secure transfer of check image files, and consolidation of credit card transactions.
The Sterling Commerce Connect:® product line provides the
  • management capability,
  • auditing features,
  • security enforcement and
  • workload balancing
that organizations need to address the inherent complexity and unreliability of networks.
Connect: products have the ability to enforce security, balance the use of network resources, and automatically recover from the interruptions that invariably occur. FTP, by comparison, does none of these.
Improve productivity
Connect:Direct provides script-based automation, scheduling and alert notifications for 24x7 unattended operations. Unlike FTP implementations, Connect:Direct eliminates the need for manual intervention in data delivery, improving the productivity of your people and the reliability of your business processes.
Gain unprecedented scalability
Event-based architecture enables high volumes and large files—with no product-defined limits on file sizes. Scalability ensures that you can handle peak demand and keep pace as your business volumes grow, whether you operate mainframes or distributed/clustered servers.
Count on reliable file delivery
An acid test for any file transfer system is how it responds when there is a failure. Connect:Direct, which also supports various clustering technologies and IBM Sysplex on the mainframe, provides built-in automation and checkpoint restart to ensure lights-out operations.
Move files with confidence
The proprietary and secure Connect:Direct protocol has never been breached. That solutions help customers satisfy regulatory and industry requirements within their file transfer operations, including compliance with Sarbanes-Oxley, Payment Card Industry (PCI) and healthcare (HIPAA) requirements. Today, Sterling Commerce is the world’s most trusted collaboration partner and the leader in secure file transfer, with over 48 percent market share.
Automation and Management
Supports 24x7 unattended operations
Schedules jobs on a one-time, recurring, or continuous basis
Assigns and manages file transfer workload
Event driven alert notification
Process language builds scripts to provide integration with back-end systems
API and SDK for programmatic access by other applications
Assured File Delivery
Support Check point restart
Automatic recovery from network interruptions
Automated alert notifications for success/failure
Security and Compliance
Interfaces with operating system security for user authentication
Provides a complete audit trail of data movement through extensive statistics logs
User authentication
x.509 certificates for authentication
Data encryption (SSL/TLS)
Certificate and Certificate Revocation List (CRL) checking
FIPS 140-2 and Common Criteria certification
Demilitarized Zone (DMZ)-based authentication, • session break and SSL termination
Ensure that no file is stored in the DMZ
No inbound ports opened in the firewall
Validation of the Connect:Direct protocol
Multiple Platform Support
z/OS, z/VM, and z/VSE
i5/OS (OS/400
UNIX and Linux
HP NonStop
Connect:Direct Select (Java version that can run on multiple platforms)
Network Protocols Support
Mirjana's picture

HTTP protocol

The Hypertext Transfer Protocol (HTTP) is an application-level protocol for distributed, collaborative, hypermedia information systems. It is a generic, stateless, protocol which can be used for many tasks beyond its use for hypertext, such as name servers and distributed object management systems, through extension of its request methods, error codes and headers. A feature of HTTP is the typing and negotiation of data representation, allowing systems to be built independently of the data being transferred.
HTTP provides the capability not only for your web browser to request pages and files from the webserver, but also HTTP provides the ability for your browser to send information back to the server. Usually this information is in the form of text box information, check boxes, and radio buttons you click on or fill out when you register on a particular website, respond to a poll, or submit any form.

HTTP characteristics

  • Stateless - Each transaction between the client and server is independent and no state is set based on a previous transaction or condition.
  • Uses requests from the client to the server and responses from the server to the client for sending and receiving data.
The HTTP protocol is a request/response protocol. A client sends a request to the server in the form of a request method, URI, and protocol version, followed by a MIME-like message containing request modifiers, client information, and possible body content over a connection with a server. The server responds with a status line, including the message's protocol version and a success or error code, followed by a MIME-like message containing server information, entity metainformation, and possible entity-body content.
Most HTTP communication is initiated by a user agent and consists of a request to be applied to a resource on some origin server. In the simplest case, this may be accomplished via a single connection between the user agent and the origin server.
A more complicated situation occurs when one or more intermediaries are present in the request/response chain. There are three common forms of intermediary: proxy, gateway, and tunnel.
A proxy is a forwarding agent, receiving requests for a URI in its absolute form, rewriting all or part of the message, and forwarding the reformatted request toward the server identified by the URI.
A gateway is a receiving agent, acting as a layer above some other server(s) and, if necessary, translating the requests to the underlying server's protocol.
A tunnel acts as a relay point between two connections without changing the messages; tunnels are used when the communication needs to pass through an intermediary (such as a firewall) even when the intermediary cannot understand the contents of the messages.
In HTTP/1.0, most implementations used a new connection for each request/response exchange. In HTTP/1.1, a connection may be used for one or more request/response exchanges, although connections may be closed for a variety of reasons.

 HTTP Methods

The HTML specifications technically define the difference between GET and POST:
GET means that form data is to be encoded (by a browser) into a URL, it creates a query string of the name-and-value pairs and then appends the query string to the URL of the script on the server that handles the request.
While the POST means that the form data is to appear within a message body, it passes the name-and-value pairs in the body of the HTTP request message


HTTP is insecure and is subject to man-in-the-middle and eavesdropping attacks which can let attackers gain access to website accounts and sensitive information. HTTPS is designed to withstand such attacks and is secure.
HTTPS can create a secure channel over an insecure network.
Mirjana's picture

SFTP protocol

SSH protocol basics

Secure Shell (SSH), sometimes known as Secure Socket Shell, is a Unix-based command interface and protocol for securely getting access to a remote computer. It is widely used by network administrators to control Web and other kinds of servers remotely. SSH is actually a suite of three utilities - slogin, ssh, and scp - that are secure versions of the earlier UNIX utilities, rlogin, rsh, and rcp.
SSH also offers useful features:
  • Compression - traffic may be optionally compressed at the stream level.
  • Public key authentication - optionally replacing password authentication.
  • Authentication of the server - making ”man-in-the-middle” attack more difficult
  • Port forwarding - arbitrary TCP sessions can be forwarded over an SSH connection.
  • X11 forwarding - SSH can forward your X11 sessions too.
  • File transfer - the SSH protocol family includes two file transfer protocols (SFTP and SCP)
The file transfer capabilities of SSH are provided by utilities included with most SSH products, typically, these utilities are called scp and sftp.
The scp and sftp utilities use the SCP and SFTP protocols (respectively) to provide file transfer capabilities and use the encrypted SSH tunnel to provide security.
SFTP stands for ‘Secure File Transfer Protocol’. The Secure File Transfer Protocol ensures that data are securely transferred using a private and safe data stream.
The SFTP protocol's main purpose is to transfer data, but it is also used to obtain general access to the SFTP server's file system. The SFTP protocol runs on a secure channel - no clear text passwords or file data are transferred.
Compared to the earlier SCP protocol, which allows only file transfers, the SFTP protocol allows for a range of operations on remote files – it is more like a remote file system protocol. An SFTP client's extra capabilities compared to an SCP client include resuming interrupted transfers, directory listings, and remote file removal.
Although both SCP and SFTP utilize the same SSH encryption during file transfer with the same general level of overhead, SCP is usually much faster than SFTP at transferring files especially on high latency networks. This happens because SCP implements a more efficient transfer algorithm, one which does not require waiting for packet confirmations. This leads to faster speed but comes at the expense of not being able to interrupt a transfer, so unlike SFTP, SCP transfer cannot be canceled without terminating the session.
The encryption used by SSH provides confidentiality and integrity of data over an insecure network, such as the Internet.

SSH protocol security

SSH commands are encrypted and secure in several ways. Both ends of the client/server connection are authenticated using a digital certificate, and passwords are protected by being encrypted.
SSH encrypts traffic in both directions, preventing traffic sniffing and password theft.
Authentication methods
(In all cases machine authentication is by a public key.)
  • Password - It provides strong protection against password sniffing and third party session monitoring, better protecting your authentication credentials and privacy.
  • Public key authentication - Client will check the identity of server.
Password authentication - Host key
Each server has a unique identifying code, called a host key. These keys prevent a server from forging another server’s key. If you connect to a server and you receive an unexpected host key, SSH client can warn you that the server may have been switched and that a spoofing attack might be underway.
Every SSH client records the host key for each server you connect to, in the configuration storage. Every time you connect to a server, it compares the server’s host key to the host key you received the last time you connected. If the keys differ, you will receive a warning and a chance to abandon your connection before you enter any private information such as a password.
However, when you connect to a server for the first time, SSH client has no way of telling whether the host key is the right one or not. So it gives the warning shown above, and asks you whether you want to trust this host key or not.
Whether or not to trust the host key is your choice.

Public key authentication

Secure Shell (SSH) public key authentication can be used by a client to access servers.
SSH includes an ability to authenticate users using public keys. Instead of authenticating the user with a password, the server will verify a challenge signed by the user’s private key against its copy of the user’s public key.
Setting up public key authentication requires you to generate a public/private key pair and install the public portion on the server. It is also possible to restrict what a given key is able to do and what addresses they are allowed to log in from.
This picture will show how to exchange keys in Password and/or Public key authentication:


Before establishing a connection, the SFTP server sends an encrypted fingerprint of its public host keys to ensure that the SFTP connection will be exchanging data with the correct server.
The first time the connection is established, this key is not yet known to the client program and must therefore be confirmed by the user before data is exchanged for the first time. Once you have established a connection to an FTP server and are sure that it is really the correct server, you should save the fingerprint information locally. This enables you to check the fingerprint information against the data you saved every time you establish a new connection to ensure that no one is between you and the server. Different servers issue fingerprints only once. They are generated by a server's private key.
Syndicate content