You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-02 - 04:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Aug 02, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19216997327,8cyclictest131rcu_preempt00:05:171
1922499631,58cyclictest0-21swapper/222:55:132
1920899593,5cyclictest131rcu_preempt21:30:100
1921699552,43cyclictest11030-21ssh22:25:131
1922499541,42cyclictest0-21swapper/222:35:162
1921699542,38cyclictest7316-21ssh22:05:121
1920899534,30cyclictest5297-21ssh21:10:130
1921699514,32cyclictest16351-21ssh22:10:141
1921699511,41cyclictest14690-21ssh22:45:131
1920899514,38cyclictest22305-21ssh23:25:130
19208995119,24cyclictest12-21ksoftirqd/023:15:140
1922499491,38cyclictest0-21swapper/221:10:152
1920899493,31cyclictest6388-21ssh00:10:130
1920899493,31cyclictest6388-21ssh00:10:120
1921699483,32cyclictest23303-21ssh21:20:121
1921699481,38cyclictest17080-21ssh23:40:131
1921699481,34cyclictest23779-21ssh22:50:141
1922499471,41cyclictest0-21swapper/222:45:142
19208994735,8cyclictest23291-21apt-get21:20:120
1923399462,30cyclictest35966irq/58-eth021:20:133
1922499461,41cyclictest0-21swapper/222:25:132
1922499461,40cyclictest19364-21apt-get19:10:102
1922499461,37cyclictest0-21swapper/220:25:112
1922499460,40cyclictest27346irq/40-dwc2_hso19:20:142
1921699462,33cyclictest3609-21ssh21:45:141
19208994631,10cyclictest26981-21ssh21:40:120
19208994617,18cyclictest12029-21cat00:30:200
19208994616,15cyclictest1-21systemd23:40:030
1923399453,26cyclictest35966irq/58-eth022:35:143
1922499452,40cyclictest0-21swapper/222:05:132
1922499451,41cyclictest18313-21apt-get20:50:132
1922499451,41cyclictest18313-21apt-get20:50:132
1921699452,36cyclictest19997-21ssh22:30:151
1920899452,36cyclictest24448-21ssh00:20:130
19208994517,11cyclictest29995-21diskstats00:05:200
1923399442,27cyclictest35966irq/58-eth020:35:123
1922499442,38cyclictest0-21swapper/223:20:152
1922499441,36cyclictest15393-21ssh00:15:142
1922499440,40cyclictest0-21swapper/220:40:142
1922499440,38cyclictest0-21swapper/222:00:142
19208994438,3cyclictest3805-21apt-get20:35:120
19208994418,9cyclictest9890-21ssh23:53:120
19208994417,7cyclictest2463-21ntp_states20:30:260
19208994416,11cyclictest11706-21ssh21:49:280
19208994416,10cyclictest19800-21ssh23:22:290
19208994415,18cyclictest15507-21diskstats00:15:150
1923399435,24cyclictest35966irq/58-eth022:50:133
1922499431,39cyclictest0-21swapper/219:35:122
19208994338,2cyclictest25423-21apt-get19:50:140
19208994330,9cyclictest12631-21apt-get21:50:110
19208994318,10cyclictest27775-21rm22:33:570
19208994317,9cyclictest14774-21ssh23:37:460
19208994316,10cyclictest26075-21ssh23:45:100
19208994315,16cyclictest2079-21ssh23:12:590
19208994315,11cyclictest32299-21cut23:30:200
19208994314,9cyclictest14416-21diskstats21:15:160
19208994313,14cyclictest3263-21kthreadcore20:00:200
1923399423,25cyclictest35966irq/58-eth000:00:133
1923399422,22cyclictest35966irq/58-eth022:15:133
1922499422,35cyclictest11810-21apt-get23:55:152
1922499421,37cyclictest0-21swapper/220:35:102
1922499421,36cyclictest0-21swapper/222:20:112
1922499421,19cyclictest2916-21grep19:25:192
1922499420,27cyclictest22279-21cut21:55:192
1921699422,34cyclictest15378-21ssh00:15:141
1921699422,33cyclictest25368-21ssh22:15:141
19208994236,2cyclictest7310-21apt-get22:05:140
19208994234,5cyclictest20744-21apt-get00:00:140
19208994219,9cyclictest17167-21tune2fs20:15:180
19208994217,10cyclictest12379-21ssh22:42:450
19208994216,10cyclictest20962-21ssh22:47:110
19208994215,10cyclictest30504-21ssh22:15:550
19208994215,10cyclictest23730-21ssh22:13:340
19208994214,15cyclictest0-21swapper/023:02:560
19208994210,19cyclictest32729-21ssh22:55:130
1923399412,25cyclictest35966irq/58-eth000:25:123
1923399412,19cyclictest0-21swapper/321:45:143
1922499411,35cyclictest0-21swapper/222:50:132
1922499410,26cyclictest2227-21cut21:05:212
1921699413,4cyclictest131rcu_preempt21:40:121
19208994116,9cyclictest8705-21ssh22:22:430
19208994116,16cyclictest13156-21/usr/sbin/munin20:40:330
19208994115,9cyclictest4027-21ssh22:38:360
19208994115,9cyclictest11852-21cut22:25:210
19208994115,11cyclictest24637-21ssh00:36:190
19208994115,10cyclictest30520-21ssh22:52:520
19208994115,10cyclictest26416-21ssh21:39:570
19208994114,10cyclictest29847-21ssh21:59:280
1922499401,35cyclictest0-21swapper/220:45:132
1922499401,35cyclictest0-21swapper/200:00:142
1922499400,23cyclictest0-21swapper/223:14:322
1921699402,24cyclictest18431-21ssh23:05:151
19208994018,10cyclictest274231cat19:15:230
19208994016,8cyclictest7040-21ssh21:28:150
19208994016,10cyclictest3418-21ssh22:00:480
19208994015,9cyclictest11754-21ssh23:55:120
19208994014,13cyclictest28871-21kthreadcore20:25:200
19208994013,12cyclictest22067-21ssh23:05:270
19208994013,11cyclictest9267-21kthreadcore20:05:200
19208994011,15cyclictest22195-21grep20:50:300
19208994011,15cyclictest22195-21grep20:50:290
1922499390,36cyclictest23297-21ssh21:20:122
1922499390,24cyclictest11863-21ssh23:18:542
19208993916,9cyclictest30295-21/usr/sbin/munin19:55:140
19208993916,8cyclictest6902-21ssh00:26:490
19208993915,9cyclictest15651-21/usr/sbin/munin20:10:290
19208993914,9cyclictest15255-21cat19:35:290
19208993914,9cyclictest15117-21grep20:45:220
19208993914,15cyclictest16736-21kthreadcore19:40:200
1922499381,34cyclictest23126-21apt-get20:55:112
1922499381,33cyclictest20002-21ssh22:30:142
1922499381,33cyclictest0-21swapper/219:50:142
1922499381,23cyclictest0-21swapper/223:30:332
1922499380,33cyclictest0-21swapper/200:05:162
19208993814,15cyclictest24019-21ntp_192.168.11419:45:230
19208993814,10cyclictest387-21/usr/sbin/munin21:05:150
1922499371,30cyclictest6368-21ssh00:10:112
1922499371,30cyclictest6368-21ssh00:10:112
1922499371,29cyclictest0-21swapper/221:30:112
1922499370,34cyclictest0-21swapper/223:25:112
1922499370,34cyclictest0-21swapper/221:45:132
1922499370,24cyclictest9805-21mii-tool23:00:182
1921699373,27cyclictest2757-21ssh23:50:131
19208993714,8cyclictest4763-21ntp_192.168.11419:25:230
19208993711,9cyclictest9569-21ntp_192.168.11419:30:220
19208993710,11cyclictest23801-21sed20:55:180
1922499364,24cyclictest24445-21grep19:45:272
1922499361,31cyclictest29823-21ssh23:45:262
19208993617,13cyclictest0-21swapper/019:10:110
19208993614,6cyclictest31425-21tr19:20:230
19233993519,11cyclictest35966irq/58-eth023:50:133
1922499350,30cyclictest27952-21apt-get21:00:132
1922499350,30cyclictest18407-21ssh23:05:132
1922499350,28cyclictest0-21swapper/200:30:022
1920899358,9cyclictest0-21swapper/019:10:010
1922499341,31cyclictest17901-21apt-get21:35:152
1922499341,29cyclictest1506-21apt-get22:20:022
1922499341,29cyclictest0-21swapper/221:15:132
1922499341,28cyclictest0-21swapper/220:30:142
1922499341,25cyclictest20594-21memory23:40:232
1922499340,22cyclictest8622-21diskmemload23:51:422
19208993413,8cyclictest22107-21awk20:20:180
1923399333,18cyclictest35966irq/58-eth023:55:153
1922499331,29cyclictest28911-21ssh00:20:302
1922499331,29cyclictest22792-21ntpq00:35:272
1922499331,28cyclictest7794-21sshd00:27:512
1922499331,28cyclictest0-21swapper/221:25:112
1922499331,28cyclictest0-21swapper/220:05:122
1922499330,30cyclictest7984-21apt-get23:35:152
1922499330,29cyclictest9319-21ntpq22:40:252
1922499330,29cyclictest30670-21grep21:40:262
1923399325,16cyclictest35966irq/58-eth021:55:143
1923399323,20cyclictest35966irq/58-eth021:50:113
19233993212,11cyclictest35966irq/58-eth022:30:143
1922499324,25cyclictest21546-21sshd20:19:032
1922499321,28cyclictest25109-21latency_hist22:15:022
1922499321,22cyclictest16369-21/usr/sbin/munin21:50:262
1922499320,29cyclictest6144-21sshd19:30:032
1922499320,28cyclictest1439-21/usr/sbin/munin19:55:272
1923399313,17cyclictest35966irq/58-eth019:45:123
1922499311,27cyclictest19089-21memory19:40:252
1922499310,28cyclictest27654-21/usr/sbin/munin19:15:272
1922499310,27cyclictest0-21swapper/220:00:122
1921699313,21cyclictest9433-21ssh23:00:121
1921699312,20cyclictest5626-21ssh22:40:121
1920899317,9cyclictest0-21swapper/021:00:260
1923399303,17cyclictest35966irq/58-eth022:45:123
1921699304,18cyclictest19127-21ssh00:35:131
1921699303,15cyclictest21-21ksoftirqd/119:30:101
1921699302,22cyclictest30739-21ssh22:00:141
1921699301,15cyclictest21-21ksoftirqd/120:20:121
1922499291,25cyclictest26322-21sshd20:21:042
1921699294,17cyclictest21-21ksoftirqd/119:50:141
19216992910,11cyclictest0-21swapper/121:37:241
1923399288,13cyclictest35966irq/58-eth019:50:133
1921699282,21cyclictest6397-21ssh00:10:131
1921699282,21cyclictest6397-21ssh00:10:131
1921699282,19cyclictest8905-21ssh21:30:111
1922499271,23cyclictest930-21snmpd20:13:242
1921699276,4cyclictest131rcu_preempt21:50:111
1923399263,15cyclictest0-21swapper/322:12:303
19233992610,12cyclictest0-21swapper/321:40:123
19233992514,8cyclictest35966irq/58-eth021:25:143
1921699254,16cyclictest21-21ksoftirqd/121:55:131
1922499241,19cyclictest19298-21latency_hist19:10:022
1921699242,15cyclictest5213-21ssh21:26:031
1921699242,14cyclictest28899-21ssh22:35:031
1923399232,15cyclictest35966irq/58-eth019:20:123
1921699239,7cyclictest21-21ksoftirqd/123:30:161
1921699232,13cyclictest27347-21ssh23:10:031
1921699231,20cyclictest11816-21ssh23:55:131
19216992311,4cyclictest0-21swapper/119:20:291
19216992311,3cyclictest8365-21munin-run20:40:011
1923399221,15cyclictest0-21swapper/322:25:143
1921699227,12cyclictest24451-1kworker/u9:121:10:301
1921699221,18cyclictest32714-21apt-get22:55:131
1921699221,17cyclictest30205-21apt-get19:55:131
1921699220,18cyclictest6320-21ls20:00:281
1923399213,15cyclictest35966irq/58-eth000:35:143
1923399212,13cyclictest0-21swapper/322:20:133
1923399211,13cyclictest0-21swapper/321:15:153
19233992110,6cyclictest35966irq/58-eth021:11:033
1921699219,4cyclictest21-21ksoftirqd/123:20:231
1921699219,3cyclictest21-21ksoftirqd/100:24:111
1921699218,3cyclictest21-21ksoftirqd/123:17:021
1921699217,4cyclictest21-21ksoftirqd/100:31:151
1921699216,10cyclictest21-21ksoftirqd/100:00:141
1921699212,15cyclictest4898-21diskmemload23:28:031
1921699211,4cyclictest0-21swapper/123:45:201
1921699211,17cyclictest4443-21ls00:25:221
1921699211,16cyclictest15517-21ls20:10:261
19216992110,4cyclictest21-21ksoftirqd/120:35:011
1923399209,8cyclictest0-21swapper/323:00:233
1923399209,8cyclictest0-21swapper/322:59:583
1923399209,6cyclictest0-21swapper/323:45:143
1923399209,6cyclictest0-21swapper/321:33:543
19216992014,3cyclictest21-21ksoftirqd/119:35:251
1921699201,3cyclictest131rcu_preempt20:40:251
1923399193,11cyclictest0-21swapper/323:22:033
1923399192,11cyclictest0-21swapper/323:25:043
1923399192,10cyclictest0-21swapper/323:30:033
1923399191,13cyclictest0-21swapper/323:40:153
19233991910,6cyclictest0-21swapper/323:14:173
19233991910,6cyclictest0-21swapper/300:15:313
1921699197,3cyclictest131rcu_preempt21:05:221
1921699194,3cyclictest131rcu_preempt19:25:201
1921699193,13cyclictest19289-21ps19:40:281
1921699192,4cyclictest131rcu_preempt23:35:161
19216991911,6cyclictest1496-21cron22:20:061
1921699191,15cyclictest29453-21kthreadcore20:25:211
1921699191,15cyclictest24237-21ls19:15:151
1921699191,13cyclictest14307-21ssh21:15:141
1921699191,10cyclictest16821-21missed_timers20:45:241
1923399188,6cyclictest35966irq/58-eth023:15:163
19233991812,5cyclictest33-21ksoftirqd/320:40:153
1923399181,13cyclictest0-21swapper/320:10:193
1923399181,13cyclictest0-21swapper/319:30:263
1923399181,11cyclictest0-21swapper/322:05:023
19233991810,6cyclictest0-21swapper/322:44:153
19233991810,6cyclictest0-21swapper/322:05:093
19233991810,6cyclictest0-21swapper/321:35:223
19233991810,6cyclictest0-21swapper/300:11:053
19233991810,6cyclictest0-21swapper/300:11:053
1921699181,9cyclictest26956-21unixbench_multi20:55:291
1921699180,7cyclictest24288-21xargs19:45:281
1923399178,6cyclictest35966irq/58-eth023:35:153
19233991711,4cyclictest0-21swapper/323:05:453
19233991711,4cyclictest0-21swapper/300:30:273
1923399171,10cyclictest0-21swapper/320:30:293
19233991710,4cyclictest33-21ksoftirqd/300:20:173
19233991710,4cyclictest0-21swapper/300:05:353
1921699178,6cyclictest21-21ksoftirqd/120:50:131
1921699178,6cyclictest21-21ksoftirqd/120:50:121
1921699174,3cyclictest0-21swapper/120:15:231
1921699173,10cyclictest0-21swapper/119:10:101
1921699171,8cyclictest0-21swapper/120:05:181
1921699171,12cyclictest30608-21kthreadcore21:00:211
1923399166,7cyclictest33-21ksoftirqd/320:25:213
1923399159,3cyclictest33-21ksoftirqd/320:56:083
1923399152,9cyclictest0-21swapper/320:50:143
1923399152,9cyclictest0-21swapper/320:50:143
1923399152,9cyclictest0-21swapper/319:10:133
1923399149,3cyclictest0-21swapper/320:20:143
1923399149,3cyclictest0-21swapper/319:35:173
1923399149,3cyclictest0-21swapper/319:15:303
1923399141,7cyclictest0-21swapper/320:05:273
1923399141,10cyclictest0-21swapper/321:05:183
1921699140,11cyclictest19236-21sort19:10:011
1923399131,7cyclictest0-21swapper/320:45:173
1923399124,5cyclictest0-21swapper/321:00:123
1923399121,9cyclictest0-21swapper/319:45:013
1923399121,6cyclictest0-21swapper/319:25:113
1923399111,8cyclictest0-21swapper/320:00:153
1923399111,7cyclictest0-21swapper/320:15:183
1923399111,7cyclictest0-21swapper/319:55:173
1923399101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional