You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-26 - 06:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri Jun 26, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1253299680,32cyclictest9323-21ssh00:15:122
1252499558,36cyclictest19139-21ssh22:15:131
12542995339,9cyclictest35966irq/58-eth021:20:133
1252499535,36cyclictest999-21ssh22:05:141
1252499523,6cyclictest131rcu_preempt19:35:131
1251699491,42cyclictest22933-21ssh22:35:140
1252499482,38cyclictest16735-21ssh21:20:131
1252499472,37cyclictest5474-21ssh23:55:131
1253299451,33cyclictest20476-21ssh21:40:122
12516994324,15cyclictest536-21usb-storage20:50:110
1251699421,10cyclictest131rcu_preempt23:00:130
1251699421,10cyclictest131rcu_preempt23:00:120
1252499410,32cyclictest13825-21ssh22:30:131
1254299403,24cyclictest35966irq/58-eth023:25:143
1251699402,31cyclictest12386-21ssh23:05:150
12532993915,15cyclictest27-21ksoftirqd/223:50:102
1251699392,33cyclictest23569-21ssh00:05:130
1251699392,31cyclictest28857-21sh23:50:120
1252499381,29cyclictest7045-21sshd23:20:141
1251699384,31cyclictest17615-21ssh22:50:140
1251699382,26cyclictest4877-21rm22:25:120
1254299371,26cyclictest35966irq/58-eth019:35:123
1252499373,27cyclictest16037-21rm23:25:121
12524993516,10cyclictest21-21ksoftirqd/120:50:121
1251699358,22cyclictest12-21ksoftirqd/021:30:120
12542993414,11cyclictest35966irq/58-eth000:35:123
1254299332,20cyclictest35966irq/58-eth023:55:113
1254299332,18cyclictest35966irq/58-eth021:40:113
1252499332,25cyclictest10063-21ssh22:10:131
1251699333,20cyclictest29578-21ssh21:45:130
12516993314,7cyclictest12-21ksoftirqd/022:40:100
1253299322,19cyclictest27-21ksoftirqd/220:30:142
12524993221,6cyclictest14583-21ssh00:00:181
1251699326,17cyclictest12-21ksoftirqd/021:00:140
12516993213,12cyclictest28041-21kworker/0:023:45:130
1254299303,15cyclictest0-21swapper/322:15:123
12542993016,9cyclictest35966irq/58-eth000:25:143
1253299303,16cyclictest27346irq/40-dwc2_hso22:10:112
1253299300,14cyclictest3321-21ssh23:00:142
1253299300,14cyclictest3321-21ssh23:00:132
1251699304,18cyclictest15195-21ssh21:55:120
1251699304,17cyclictest30466-21ssh23:15:120
12516992914,9cyclictest12-21ksoftirqd/020:20:110
1253299282,17cyclictest27346irq/40-dwc2_hso22:45:122
1253299281,19cyclictest27398-21ssh00:25:152
1251699285,13cyclictest24337-21ssh22:00:130
1254299279,10cyclictest35966irq/58-eth022:25:133
12542992717,7cyclictest35966irq/58-eth023:30:113
12532992710,13cyclictest0-21swapper/223:45:132
1251699272,20cyclictest10720-21sshd23:40:130
1251699272,19cyclictest9092-21apt-get19:40:110
1251699271,21cyclictest27119-21apt-get19:25:120
1254299262,14cyclictest35966irq/58-eth021:15:133
1254299262,13cyclictest35966irq/58-eth022:30:123
12542992615,8cyclictest35966irq/58-eth022:10:143
1253299262,9cyclictest8185-21ssh00:14:142
1253299262,7cyclictest0-21swapper/223:55:122
1251699263,17cyclictest25061-21ssh23:30:100
1251699263,15cyclictest26701-21ssh22:55:150
1251699262,18cyclictest9339-21sh00:15:130
1251699262,17cyclictest4114-21ssh00:30:140
1251699261,20cyclictest31083-21sh21:10:140
1251699261,19cyclictest5555-21ls23:55:130
1253299252,18cyclictest6208-21ssh21:50:122
1253299251,17cyclictest31897-21find22:40:102
1251699252,4cyclictest131rcu_preempt00:35:130
1251699251,19cyclictest27377-21sshd00:25:120
1251699251,19cyclictest11249-21ssh21:15:240
1254299249,10cyclictest35966irq/58-eth000:05:143
1253299247,8cyclictest27736-21grep00:05:322
1253299243,18cyclictest12510-21cyclictest23:20:132
1253299241,13cyclictest0-21swapper/222:15:132
1253299240,17cyclictest31084-21ssh21:10:142
1252499242,16cyclictest7687-21ssh21:15:141
1251699242,17cyclictest32642-21ssh00:10:150
1253299232,4cyclictest0-21swapper/222:25:122
1253299231,5cyclictest22047-21ls19:50:242
1253299231,18cyclictest22804-21tr23:10:212
1253299231,16cyclictest27346irq/40-dwc2_hso00:20:152
1253299231,14cyclictest4578-21unixbench_singl20:05:272
1253299231,13cyclictest23020-21ssh22:35:152
1253299231,10cyclictest17893-21ls22:30:302
1252499232,7cyclictest21-21ksoftirqd/100:30:321
12524992318,3cyclictest536-21usb-storage22:55:041
1251699238,5cyclictest131rcu_preempt22:20:140
1251699238,5cyclictest131rcu_preempt22:20:130
1251699235,12cyclictest21444-21ssh23:10:130
1253299229,10cyclictest12335-21acpi23:05:132
1253299228,11cyclictest31975-21switchtime20:00:272
1253299227,13cyclictest11445-21latency_hist20:50:022
1253299226,13cyclictest1-21systemd00:03:172
1253299221,17cyclictest19657-21kthreadcore19:15:212
1253299221,15cyclictest131rcu_preempt00:30:202
1252499227,11cyclictest21-21ksoftirqd/121:40:121
1252499223,15cyclictest13119-21ssh00:35:131
1252499222,17cyclictest27397-21ssh00:25:151
1252499221,5cyclictest131rcu_preempt22:20:151
1252499221,5cyclictest131rcu_preempt22:20:141
1252499221,18cyclictest21509-21cpuspeed23:10:151
1252499221,17cyclictest27140-21missed_timers00:05:261
1252499221,11cyclictest25917-21latency_hist21:05:021
1251699224,9cyclictest0-21swapper/020:35:130
1251699224,9cyclictest0-21swapper/020:35:120
1251699221,18cyclictest11448-21ls20:50:020
1251699221,16cyclictest18848-21cut19:50:160
12516992210,9cyclictest5507-21sshd20:10:130
1254299211,9cyclictest35966irq/58-eth020:05:133
12542992112,4cyclictest0-21swapper/322:55:333
1253299216,6cyclictest0-21swapper/220:20:222
1253299216,10cyclictest0-21swapper/220:50:142
1253299215,14cyclictest32426-21head22:20:322
1253299215,14cyclictest32426-21head22:20:322
1253299215,13cyclictest16067-21cut21:55:202
1253299211,6cyclictest30678-21diskmemload21:22:362
1253299211,16cyclictest27074-21grep23:30:192
1253299211,16cyclictest20497-21irqrtprio20:25:172
1253299211,15cyclictest27346irq/40-dwc2_hso21:35:132
1253299211,11cyclictest30481-21ssh23:15:132
1253299211,11cyclictest14147-21idleruntime19:45:162
1253299211,10cyclictest12349-21wc19:40:202
12532992110,9cyclictest16360-21diskmemload21:20:012
12524992113,5cyclictest17580-21/usr/sbin/munin22:50:291
1252499210,16cyclictest30361-21ssh21:45:201
1251699212,9cyclictest0-21swapper/020:05:130
1251699211,16cyclictest930-21snmpd21:25:130
1251699211,15cyclictest167-21systemd-network20:25:120
12516992110,9cyclictest14572-21apt-get00:00:180
1254299207,9cyclictest0-21swapper/323:40:143
1254299202,10cyclictest35966irq/58-eth021:45:143
12542992012,6cyclictest0-21swapper/323:10:213
1253299207,10cyclictest29425-21ntpq21:25:252
1253299205,5cyclictest10730-21irqcore20:15:172
1253299204,6cyclictest2098-21ps23:35:182
1253299203,10cyclictest4621-21ps19:35:192
1253299202,9cyclictest23429-21kthreadcore19:20:182
1253299201,9cyclictest2206-21wc20:40:142
1253299201,9cyclictest20157-21grep23:25:302
1253299201,9cyclictest17226-21kthreadcore20:55:182
1253299201,9cyclictest13298-21ssh00:35:162
12532992014,4cyclictest27-21ksoftirqd/221:30:222
12532992013,5cyclictest0-21swapper/220:35:142
12532992013,5cyclictest0-21swapper/220:35:132
1253299201,17cyclictest29783-21fschecks_count21:45:172
1253299201,16cyclictest7815-21kthreadcore20:10:212
1253299201,16cyclictest30920-21grep19:25:282
1253299201,16cyclictest26690-21apt-get22:55:152
1253299201,16cyclictest18236-21irqrtprio22:50:192
1253299201,15cyclictest21439-21fschecks_count21:00:152
1253299201,11cyclictest27150-21ssh22:00:232
1253299201,11cyclictest13147-21kthreadcore23:40:202
1253299201,10cyclictest12642-21grep19:10:022
1253299200,15cyclictest17389-21ls19:15:022
1252499207,10cyclictest6356-21/usr/sbin/munin21:50:161
1252499202,15cyclictest18937-21ntp_states21:55:261
1252499201,16cyclictest2374-21ssh21:30:111
12524992011,6cyclictest11360-21cron20:50:011
1252499200,17cyclictest23122-21uniq22:35:161
1251699208,8cyclictest22906-21awk22:15:270
1251699202,12cyclictest0-21swapper/020:55:130
12516992013,4cyclictest10777-21processes23:20:260
12516992010,8cyclictest23837-21tail19:55:160
12516992010,7cyclictest18069-21kthreadcore23:25:230
1254299198,9cyclictest0-21swapper/323:08:323
1254299198,6cyclictest0-21swapper/321:25:203
1254299192,13cyclictest0-21swapper/323:20:143
1254299192,11cyclictest0-21swapper/320:50:053
1254299191,13cyclictest0-21swapper/319:45:143
1254299191,12cyclictest0-21swapper/321:10:013
12542991910,6cyclictest35966irq/58-eth022:05:153
1253299191,9cyclictest1152-21fschecks_count22:05:182
1253299191,10cyclictest31977-21cpuspeed19:30:122
1253299191,10cyclictest27330-21kthreadcore21:05:212
1253299191,10cyclictest23638-21fschecks_count19:55:162
1252499199,7cyclictest10933-21kthreadcore22:45:221
1252499198,8cyclictest24671-21cat20:30:021
1252499197,9cyclictest8141-21kthreadcore20:10:201
1252499196,9cyclictest0-21swapper/100:20:141
1252499196,4cyclictest29726-21kthreadcore23:50:181
1252499193,12cyclictest26253-21tr23:30:191
1252499192,8cyclictest0-21swapper/123:44:501
1252499191,8cyclictest22343-21wc19:50:271
12524991911,6cyclictest31888-21/usr/sbin/munin22:40:291
12524991910,6cyclictest18448-21sshd19:47:521
1252499190,13cyclictest27223-21timerwakeupswit19:55:281
1251699194,6cyclictest12-21ksoftirqd/022:45:220
1251699193,9cyclictest0-21swapper/021:05:130
1251699192,12cyclictest14947-21memory21:35:230
1251699191,3cyclictest16706-21apt-get21:20:130
1251699191,14cyclictest22066-21awk00:20:270
1251699191,10cyclictest0-21swapper/022:08:010
1254299189,6cyclictest35966irq/58-eth000:30:153
1254299189,6cyclictest0-21swapper/322:02:303
1254299189,6cyclictest0-21swapper/321:11:393
1254299188,6cyclictest35966irq/58-eth021:35:133
1254299187,9cyclictest0-21swapper/300:15:223
1254299184,5cyclictest35966irq/58-eth023:35:113
1254299182,10cyclictest0-21swapper/323:50:053
12542991810,6cyclictest0-21swapper/321:54:183
12542991810,3cyclictest0-21swapper/322:20:083
12542991810,3cyclictest0-21swapper/322:20:073
1252499189,6cyclictest0-21swapper/123:15:171
1252499184,11cyclictest2458-21kthreadcore23:35:191
1252499183,12cyclictest9018-21grep22:25:301
1252499183,12cyclictest25597-21grep19:20:221
1252499182,6cyclictest28025-21/usr/sbin/munin22:00:281
1252499182,13cyclictest19668-21kthreadcore19:15:201
1252499182,12cyclictest30077-21grep20:35:191
1252499182,12cyclictest30077-21grep20:35:181
12524991812,4cyclictest21-21ksoftirqd/119:42:511
1252499181,13cyclictest12985-21ntp_192.168.11400:15:261
1252499181,13cyclictest11405-21ssh21:35:131
1252499180,9cyclictest18936-21kthreadcore20:55:191
1251699184,10cyclictest0-21swapper/021:50:130
1251699183,11cyclictest4637-21kthreadcore23:35:220
12516991811,5cyclictest11908-21tr22:10:200
1251699181,13cyclictest24766-21sshd20:30:130
12516991810,5cyclictest14404-21irqrtprio22:30:180
1251699180,14cyclictest14146-21ls19:45:150
1254299179,6cyclictest35966irq/58-eth000:00:183
1254299179,6cyclictest0-21swapper/323:46:453
1254299179,6cyclictest0-21swapper/322:52:023
1254299179,6cyclictest0-21swapper/322:46:043
1254299179,6cyclictest0-21swapper/322:40:053
1254299179,6cyclictest0-21swapper/300:21:383
1254299179,6cyclictest0-21swapper/300:10:363
1254299179,3cyclictest0-21swapper/321:31:023
1254299178,6cyclictest0-21swapper/323:15:263
1254299178,6cyclictest0-21swapper/322:35:043
12542991711,4cyclictest0-21swapper/323:02:183
12542991711,4cyclictest0-21swapper/323:02:173
1254299171,12cyclictest0-21swapper/321:00:163
1254299171,11cyclictest0-21swapper/319:55:023
1252499179,6cyclictest17978-1kworker/u9:121:13:041
1252499177,6cyclictest4864-21kthreadcore23:00:201
1252499177,6cyclictest4864-21kthreadcore23:00:201
1252499176,8cyclictest622-21sed00:10:181
1252499176,7cyclictest21-21ksoftirqd/119:25:101
1252499171,8cyclictest1353-21grep19:30:191
1252499171,12cyclictest28304-21apt-get20:00:111
1252499171,12cyclictest26345-21irqrtprio21:25:191
1252499171,12cyclictest16202-21kthreadcore20:20:171
1252499171,12cyclictest10660-21grep20:15:181
1252499171,11cyclictest4490-21cut20:40:201
12524991710,4cyclictest20880-21kthreadcore23:45:201
12524991710,4cyclictest12942-21irqrtprio23:05:181
1252499170,14cyclictest3869-21tr20:05:221
1251699173,11cyclictest5028-21tr19:35:200
1251699172,12cyclictest17572-21diskstats19:15:150
1251699172,10cyclictest0-21swapper/020:15:140
1251699171,7cyclictest13852-21tr19:10:180
1251699171,13cyclictest25725-21ntp_192.168.11419:20:230
1251699171,12cyclictest6557-21apt20:45:000
1251699171,12cyclictest31997-21diskstats19:30:140
1251699171,12cyclictest20445-21ssh21:40:100
1254299168,6cyclictest0-21swapper/321:55:033
1252499162,11cyclictest24877-21fschecks_count20:30:151
1252499162,11cyclictest13712-21tr19:10:191
1252499161,11cyclictest26486-21sed21:05:191
1251699164,9cyclictest31588-21ls20:00:220
1254299155,6cyclictest35966irq/58-eth020:25:123
1251699152,10cyclictest12611-21latency_hist19:10:020
1254299149,3cyclictest0-21swapper/320:40:133
1254299149,3cyclictest0-21swapper/320:00:283
1254299149,3cyclictest0-21swapper/319:55:213
1254299141,6cyclictest0-21swapper/320:30:223
1254299141,4cyclictest0-21swapper/320:20:243
1254299141,10cyclictest0-21swapper/320:10:173
1252499141,10cyclictest12563-21cat19:10:011
1254299138,3cyclictest0-21swapper/320:49:233
1254299138,3cyclictest0-21swapper/320:18:143
1254299135,6cyclictest0-21swapper/319:40:193
1254299131,10cyclictest0-21swapper/320:35:293
1254299131,10cyclictest0-21swapper/320:35:293
1254299126,4cyclictest0-21swapper/319:10:263
1254299121,4cyclictest0-21swapper/319:20:193
1254299112,5cyclictest0-21swapper/319:25:123
1254299111,7cyclictest0-21swapper/320:55:263
1254299101,7cyclictest0-21swapper/319:15:133
1254299101,6cyclictest0-21swapper/319:30:133
1254299101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional